Friday, September 7, 2012

20 thoughts on skill vs. chance in poker, part 3: Outcome-based approaches fail on populations of similarly-skilled players

<-- Part 2: Richard Garfield on luck in games

The vast majority of approaches to predominance in poker tend to rely upon empirical evidence of outcomes from real or simulated poker games. Examples include studies that look at the relationship at the winrates and outperformance of superior players in a game such as Steven Levitt's paper on the topic, studies that look at the relationship between a player’s expected value and variance over time such as the critical repetition frequency of the work of Fiedler and Rock, and all other studies that rely on data from real play from laboratory experiments or online poker databases. These outcome-based approaches tend to be practical in that they appeal to the layperson's intuition; a game of only chance would mean that each player has an equal chance of winning, so a game where the superior player wins only slightly more often than that could be seen as "mostly chance", but if the better player wins close enough to 100% of the time then the game is surely "mostly skill". The linked studies are both great and many studies of this type are worthy contributions to the cause, but we should take caution and remember that such approaches are entirely dependent on the different levels of skill within the sample population.

If all players being observed are of approximately the same skill, then they will employ approximately-identical strategies and any outcome-based approach will be unable to distinguish ANY symmetric multiplayer game from pure chance.

For example, outcomes in chess, the quintessential game of skill, will be identical to outcomes of coinflips when the game is played between similarly-skilled players, whether they be similarly-ranked amateurs or superhumans executing the Nash equilibrium strategy for chess.

As if potentially deeming chess to be a game of predominantly chance weren't concerning enough, an outcome-based approach to predominance based on data from real-world play will also allow the possibility of changing its classification of a game as time goes on.

For a poker example, let's say that any particular outcome-based approach to predominance finds that $1/$2 NL cash games are just barely predominantly skill. Then the same outcome-based approach will likely conclude that $10/$20 NL cash games are predominantly chance, since the edges between players and the differences between winrates are much lower at the highest levels of competition (while the variances likely stay the same or increase). Yet, nothing in the definition of the game nor its fundamental nature has changed! Paradoxically, this convergence of winrates is of course due to the fact that the game strategy is more developed, that the depths of skill in the game have been further explored, and that the players who have reached the top levels of the game have become better players along the way. The game outcomes couldn't change between different levels of play unless the game was one with a tremendous amount of skill.

Outcome-based approaches to predominance can identify skill when observing an expert playing against a weaker player, but cannot distinguish strategy games from coinflips when observing players of similar skill levels.

Theoretically, the factors of skill and chance within a game cannot vary based on the particular players that happen to be playing it at any given time. Skill and chance are inherent to the structure of the game. They exist regardless of whether or not any given player properly taps into and executes the skill. If two naïve players were to play a game of skill where both of them moved randomly or without thinking at all, the factors of skill are still present in the game, it is merely that the players have decided not to use them. If somebody buys a smartphone and only ever uses it for phone calls without exploring the rest of its capabilities, that doesn't change the fact that the device is a smartphone.

While empirical evidence that some players with some strategies win a two-player game much more than half of the time does imply that the game has a skill component, the converse is not true. A game where players each win only about half of the time merely implies that the players under observation are employing nearly-identical strategies. Note that the case of a game of pure chance is a trivial instance of this, as strategies will always be identical in a game where outcomes are not affected by strategic decisions.

If, in some sudden singularity of global disinterest in poker, all but the best 100 poker players in the world decided to quit the game forever, then we're not going to be seeing the best players at the WSOP outperforming each other by as much as Levitt found using current data and the necessary time threshold for reaching the critical repetition frequency will have increased dramatically. Yet poker itself, unchanged, will still be the same game with the same skill and the same chance elements that it has in the real world.

When an outcome-based approach finds that player skill does sufficiently impact outcomes, then this is indeed evidence of skill in the game, but if such an approach fails to find this evidence, it does not imply that the game doesn't have a depth of skill — and laymen may be eager to draw this conclusion. Therefore we should be hesitant to rely solely upon such arguments.

Part 4: What if an outcome-based approach specifies observing "average" players? -->

(back to index)


  1. Wow, that was a lot of words, big words, to say "the edge of one skilled player over a similarly skilled player is negligible, while the edge of one skilled player over an unskilled player is vast" :)

    But what an important point that must be made in the argument of skill vs chance.

    Great post!

    1. Nobody has ever accused me of being too succinct on this blog ;)


Note: Only a member of this blog may post a comment.

Related Posts Plugin for WordPress, Blogger...