CSCE475/875 Multiagent Systems

Handout 12: Game Day 2 Voting Day Analysis (Amended)

October 13, 2011

List of Movies

	TITLE
1	Avatar
2	Blind Side, the
3	Dark Knight, the
4	Fast & The Furious, the
5	Finding Nemo
6	Forrest Gump
7	Gladiator
8	Hangover, the
9	Inception
10	Incredibles, the
11	Independence Day
12	Indiana Jones and the Last Crusade
13	Iron Man
14	Matrix, the
15	Meet the Parents
16	Monsters, Inc.
17	Pursuit of Happiness, the
18	Rush Hour
19	Shrek
20	Sixth Sense, the
21	Spiderman
22	Star Wars: Episode III
23	Superman Returns
24	Yes Man

Table 1. List of movies used.

Team Statistics

To compute the first set of statistics, I simply noted the time your e-mails on the winners were received. If it was received at 9:38 a.m., then 38 was recorded. If it was received at 10:11 a.m., then 11 was recorded. Since the range of time for each round never crossed from before 60 to after 60 (before 00 to after 00), it was okay to simply sum these for a total as an indicator of how fast each team was able to process the votes. Table 2 shows the results.

Team Name	R1	R2	R3	R5	R6	TOTAL
DJ Carpet	48	1	13	36	43	141
Triple Threat	48	2	14	34	43	141
Split Second	47	0	13	32	51	143
Free Agents	47	1	14	34	51	147
ULM	47	0	14	32	41	134
Power Agent	46	0	12	31	50	139
SIB	49	2	15	39	47	152
JRL	48	1	14	39	43	145
Reagent	47	1	15	33	38	134
Wolfpack	47	1	13	36	46	143

Table 2. Pseudo-time-stamps for each e-mail on winner, and the total tally in terms of “minutes”.

As shown in Table 2, ULM and Reagent were the fastest teams, followed by Power Agent, DJ Carpet & Tripe Threat, Wolfpack & Split Second, JRL, Free Agents, and finally SIB. Note that three teams turned in a different R6 preference ordering and/or winners when they submitted their Report. Those R6 preference orderings I disregarded. I only used the ones e-mailed to me during the game day, and thus the “pseudo time stamps” I used corresponded to those, not at 2:30 p.m., which would have added about 240 mins to their totals above.

Several teams did not follow instructions properly and also did not vote consistently. For example, if in Round 2, you gave 5 votes to A, 3 votes to B, and 0 votes to the rest, then in Round 3, none of the 0-vote-getters in Round 2 should receive any vote! (Also, in this case, different teams have two different approaches. One approach is to assign 1 vote to all vote getters in Round 2. Another is to assign 1 vote to only the top vote getters in Round 2. Both approaches are considered consistent.) SIB re-voted in Round 4 even though they should not since their movie was not eliminated. Also, suppose there are N 0-vote-getters in your Round 2 voting, you should not give x of those 0-vote-getters one vote while the other N-x 0-vote-getters no vote in Round 3. That is not consistent. The reason is that Cumulative Voting and Approval Voting are not complete ordering mechanisms (and they are not strict ordering mechanisms either). That means if you didn’t prefer movie A, B, and C in Round 2, you should not suddenly prefer say, A over B and C in Round 3. Further, three teams did not turn in the correct elimination round-by-round winners and also the overall winner in Round 6. There was also a team that used 24 instead of 23 in assigning the votes in Round 6. The same team also did it in an opposite manner: instead of giving 23 to the top-ranked movie, they gave 1; and so on. That actually messed up the aggregate voting results non-trivially.

Because I scored each team using “minutes”, the penalty imposed on teams who made mistakes discussed above was simply to add more “minutes” to your Game Day score. Table 3 shows the penalties and Game Day score. For incorrect elimination rounds entries, I added between 10-30 minutes depending on the number of incorrect entries. If a team didn’t submit their final elimination round tables, then I added 40 minutes.

Team Name	Speed TOTAL	R1	R2	R3	R4	R5	R6	TOTAL
Free Agents	141							141
DJ Carpet	139						10	149
Triple Threat	141						10	151
Split Second	143						10	153
ULM	134			20				154
JRL	134			5			20	159
Power Agent	147						20	167
SIB	145				10		30	185
Reagent	152					30	20	202
Wolfpack	143			20		30	10	203

Table 3. Total tally in “mins” and also penalties in terms of added pseudo-mins, and the final total.

Based on the above, Free Agents won the Game Day, followed closely by DJ Carpet, Triplet Threat, Split Second, and ULM. Wolfpack finished last because of their key mistakes in voting. Note that the teams that did not make any mistakes were ranked higher than the teams that made mistakes.

Table 4 below shows the elimination rounds and also given the voting results of Round 5 (Borda voting) the winner and loser of each pair-wise elimination.

Round 1	Round 2	Round 3	Round 4	Round 5	Round 6
Gladiator	Dark Knight, the	Dark Knight, the	Dark Knight, the	Dark Knight, the	Dark Knight, the
Dark Knight, the
Hangover, the	Hangover, the
Yes Man
Shrek	Shrek	Independence Day
Superman Returns
Independence Day	Independence Day
Sixth Sense, the
Finding Nemo	Iron Man	Matrix, the	Inception
Iron Man
Avatar	Matrix, the
Matrix, the
Star Wars: Episode III	Meet the Parents	Inception
Meet the Parents
Inception	Inception
Monsters, Inc.
Spiderman	Spiderman	Spiderman	Forest Gump
Blind Side, the
Fast and The Furious, the	Rush Hour
Rush Hour
Incredibles, the	Indiana Jones and the Last Crusade	Forrest Gump
Indiana Jones and the Last Crusade
Forrest Gump	Forrest Gump
Pursuit of Happiness, the

Table 4. Elimination rounds and winners and losers.

In general, most teams were able to follow instructions of the voting mechanisms and rules of the Game Day. But, judging from the time-stamps of the winner submissions, some teams were not as well prepared as others. For Round 6’s preference ordering, the Game Day Monitors (Rafael Leano and myself) were able to come up with the winners and losers of the elimination rounds rather quickly while some teams took quite a while in order to fill out the elimination rounds.

Rounds 1-4 are non-ranking voting mechanisms: plurality, cumulative, approval, and plurality with elimination. It is called non-ranking because we don’t necessarily need to order all candidates. In fact, there is no strict preference ordering with these voting mechanisms. Please remember that. Borda voting, on the other hand, is a ranking mechanism where one is required to provide a strict preference ordering completely.

Finally, as discussed in class, our Voting Day as preference aggregation did not motivate teams to be strategic. However, in order to win the Game Day, each team must be organized, effective, and efficient. This would be where pre-game strategies played a role—preparation of computation, understanding of the voting mechanisms, and thoughtfulness in answering the four questions.

Question Analysis

There were four questions posed.

Question 1. Using the above aggregated preference ordering, revisit Round 4 results, is the Condorcet condition satisfied? (Justify your answer.)

This condition states that if there exists a candidate such that for all other candidates at least half the voters prefer to , then must be chosen. From the textbook we know that:

Definition 9.2.3 (Condorcet winner) An outcome is a Condorcet winner

As a result of Round 5 voting (Borda voting), the Condorcet winner is the movie “Dark Knight, the”. However, if we look at Round 4 voting results (Plurality with elimination), the movie was eliminated because it only received one vote in the first round of Plurality voting.

So, Round 4’s voting mechanism did not satisfy the Condorcet condition.

Most teams (7 out of 10) understood this concept correctly.

Note that whether the Condorcet condition is satisfied is only relevant to a voting mechanism, not to a particular candidate. Thus, statements such as “Matrix satisfies the Condorcet condition over Avatar … Matrix fails the Condorcet condition over Dark Knight …” are not sensible.

Question 2. Given the Borda voting results, is there a spoiler item such that its removal from the list would cause significant changes to the preference ordering? (Justify your answer.)

First of all, removing a candidate from the list does NOT mean that all the points that the candidate has go to the pool of remaining candidates.

From our textbook and lecture:

Sensitivity to a losing candidate

Consider the following preferences by 100 agents.

Plurality would pick candidate as the winner, as would Borda. (To confirm the latter claim, observe that Borda assigns , , and the scores 103, 98, and 99 respectively.) However, if the candidate did not exist, then plurality would pick , as would Borda. (With only two candidates, Borda is equivalent to plurality.) A third candidate who stands no chance of being selected can thus act as a “spoiler,” changing the selected outcome.

So the question is looking for whether removing a “spoiler” would change the selected outcome. The winner of the Borda voting was “Dark Knight, the”. Could another movie be removed from the list such that “Dark Knight, the” lost its position after re-tallying of the voting points?

To thoroughly check for a spoiler, let “Dark Knight, the” be. First, sort the preference ordering. Remove the second place vote-getter (“Gladiator”) from the list. Then update the voting points. From Team 1, since “Gladiator” was given 20 points in the first place, the teams that received 23, 22, and 21 points all must be deducted by one point: 22, 21, and 20, respectively. Update the voting points for all teams in the same manner. Re-total the voting points. In this case, “Dark Knight, the” still received the most points. So, in this case, we proved that “Gladiator” was not a spoiler. Now, restore all voting points. Then, remove the third place vote-getter (“Inception”) and repeat the same process. And, if “Dark Knight, the” lost its first place position in any of this, then yes, there was a spoiler!

Two teams provided an interesting logic to this. They noticed that “Dark Knight, the” won by 11 voting points, and thus a spoiler could affect the outcome if and only if its removal would cost a net gain of more than 10 for another candidate to overtake “Dark Knight, the”. And there were just too few possibilities for another candidate to gain that many points.

One team offered a very interesting way to look for spoilers. “The spoilers can be found easily by finding the variance between the preferences for each movie.” In general, “movies with low preference but a high variance in preferences influence the total preference ordering the most.” But they went on to frame the spoiler question in terms of “which team is a spoiler” instead of “which movie is a spoiler.”

In our case, there was no spoiler.

Of course the main reason was due to the voting, the preferences from the teams. But, were there other factors in this MAS environment that made the chance of having a spoiler very unlikely? Yes, there were two factors. First, the larger the candidate pool, the less likely it is to have a spoiler. This is because the voting points’ differentials become less significant when there are more candidates. For example, a candidate getting a 3 and another getting a 1 in a pool of four candidates has a stronger advantage comparing to a candidate getting a 23 and another getting a 21 in a pool of twenty four candidates. This means that removing one candidate from the pool would impact a pool of four candidates more significantly than it would a pool of twenty four candidates. Second, the cluster of a few candidates as the top vote-getters, e.g., “Avatar”, “Dark Knight, the”, “Gladiator”, “Matrix”, etc. rendered the lower-ranked candidates non-consequential—they wouldn’t be able to make it to the top no matter what. So that reduces the likelihood of having a spoiler.

Most teams identified “spoilers” that caused minor changes in the preference ordering. But remember, a spoiler is supposed to change the outcome, in this case, the social choice outcome—which is the winner.

Question 3. Did the above pairwise elimination order cause an item that Pareto-dominates another candidate to finish behind the dominated candidate? (Justify your answer.)

When an item A Pareto-dominates another item B, that means at least one agent strictly prefers A over B while the other agents weakly prefers A over B. Because one team turned in a reversed-preference ordering in Round 5, basically, it ruined the chance for such an occurrence. As pointed out by the team who made the mistake, “If our correct voting would have been used, then Gladiator would have Pareto-dominated Meet the Parents” but finished behind. That was unfortunate.

Five teams did not correctly understand the meaning of Pareto domination. Most assumed that as long as most teams preferred A over B, then A Pareto dominates B. That assumption is incorrect.

Five teams correctly understood the meaning of Pareto domination.

Question 4. Provide another pairwise elimination order that would cause an item that Pareto-dominates another candidate to finish behind the dominated candidate?

Because of the problem alluded to in Question 3, this Question 4 was also rendered meaningless as well.

I had set up Questions 3 and 4 expecting that if the response to Question 3 didn’t reveal such an occurrence, then response to Question 4 would do the trick. It would have been a better learning experience for all.

Individual Team Analysis

Table 5 documents my comments on each team’s worksheet and reports.

Team Name	Pre-Game	Tracking	Mid-Game/Post-Game
Free Agents	OK.	OK.	Pointed out the bottleneck with the Game Day Monitors, also lack of clarity in file format ;They correctly pointed out the impact of the reversed-order of the teams who voted that way. Observations okay. Answered questions quite well.
Split Second	Provided a long narration of the six mechanisms – which was not appropriate to include in the pre-game strategy	OK.	Very comprehensive. A lot of good insights with many good research questions and good observations. Proposed the idea for study that the different voting systems have different expressive powers, and are thus more suitable to certain types of environments over others. (I will include the concluding paragraph later as Lessons Learned.) Did not answer questions well.
Power Agent	Quite extensive. Decided to simply just rank the movies based on the order that they appear in the excel file; Correctly pointed out how to do the computations for Round 6.	OK.	OK. Answered questions well. Observed mistakes made by other teams.
DJ Carpet	OK.	OK	Answered questions well. “Spoiler would likely occur when the number of candidates is small and votes are close together.” Did not provide post-game lessons learned.
ULM	OK.	Didn’t follow the Game Day package.	They observed mid-game the excel file received from the Monitor was convenient. Answered questions well. Noticed significant clustering in Round 2. Observed the mistake made by the Monitor in Round 5. And other mistakes by other teams. And pointed out the bottleneck caused by the Monitor.
Reagent	OK.	OK.	No mid-game/post-game observations/lessons learned. Did not answer questions well.
Triple Threat	Planned to change voting “so that our favorite second or third will win because our first pick wasn’t popular with everyone.”; that’s not consistent.	OK.	Observed mistakes in other teams. Also concluded that “Our hypothesis from this experiment is the more voters you have the less you need borda.” Did not answer questions well.
Wolfpack	OK.	OK.	Post-game observations were specific, not generalized. Answered questions well.
SIB	OK. A bit off-target.	Missing voting information	Some mid-game observations. Post-game observations were specific, not generalized. Did not answer questions well.
JRL	OK.	OK.	Post-game observations were minimal. Pointed out mistakes by other teams. Did not answer questions well.

Table 5. My comments and observations of team strategies, worksheets, and reports.

Lessons Learned

Here are some overall lessons learned.

1. From Team SplitSecond:

“In conclusion, we propose the idea for study that the different voting systems have different expressive powers, and are thus more suitable to certain types of environments over others. We hypothesize that if the environment causes agents to have strong preferences, and it is in the interest of the system to meet as many of the strong preferences as possible, Plurality voting or Cumulative Voting would be a more effective strategy than the other methods. If on the other hand, many agents have an even preference for multiple candidates we suspect that Approval Voting may offer the most system wide utility. Furthermore, we suspect that a highly distributed preference ordering would be best served with Borda Voting, as it inherently assumes such an arrangement. We also think it would be interesting to study the differences in system utility between Plurality with Elimination and Borda Voting in cases where agents strongly prefer a small subset of the total field of candidates. We further believe that studying the effect individual agents with strong preferences have on the utility of decisions made with Cumulative Voting would be an insightful exercise, that could illustrate the costs and benefits of such actors. Finally, we propose that Pairwise Elimination could address its deterministic nature, which seems harmful in the absence of a Condorcet winner, by randomly generating multiple schedules, executing them, and picking the candidate with the most wins.”

“All these propositions should also take into account the computational expense of each method. It is likely that environments which demand quick computation time would benefit more from one of the three plurality methods which do not involve elimination; whereas systems that have more flexibility in this area should also consider employing the other three more expensive methods.”

2. On computing for the winner in Round 6, several teams reported that they actually did pairwise comparison for the 23 pairs, and since there were 10 preferences to look at per pair, it took them a long while to accomplish this. That was correct. If there was a tie, then the team decided to use Borda voting counts to break the tie. That was also acceptable. Other teams who used the Borda voting would still get the correct winner. But here is one example of discrepancy: The pairwise elimination round between “Spiderman” and “Blind Side, the”. Using the actual pairwise by comparing the head-to-head preferences, “Spiderman” won. Using the Borda voting counts, “Blind Side, the” won. So, be careful about this.

3. Teams should understand the following concepts better: the Condorcet condition, the Condorcet winner, Pareto domination, spoiler, and the various voting mechanisms.

4. The Game Day monitor made a mistake. A team’s preference ordering was replaced by mistake with another team’s preference ordering. This hints at the inadequacy of how the monitor collated the votes. In the future, a better system should be in place to support the process.

5. Some teams were faster in response than some others. Think about real-time constraints in a competitive multiagent environment. Agents that are faster will enjoy an advantage. Remember this experience if and when you need to design a real-time MAS.

6. Teams that were careful were ranked higher. As a MAS designer or as an agent, being careful is a good trait to have.

7. Teams that were prepared were ranked higher. As an agent, each team should be observant, adaptive, responsive, and reflective.

Game Day League

Here is the League Standings. So far, Free Agents have won the first two Game Days.

Team Name	Learning Day	Voting Day	Auction Day	League Standings
Free Agents	1	1		2
DJ Carpet	3	2		5
Split Second	3	4		7
Power Agent	2	7		9
Triple Threat	7	3		10
ULM	5	5		10
Reagent	6	9		15
JRL	10	6		16
SIB	9	8		17
Wolfpack	7	10		17