|
|
CXP-Powered I-MINDS Capabilities 4.0 |
1.1. Introduction
IMINDS
Capability (CXP+I-MINDS) is an online collaboration system intelligently
supported by a multiagent system integrated into the ConferenceXP
framework designed by Microsoft Research.
It can be used to support groups of employees in corporate environments,
as well as students in educational settings by serving as a computer supported,
collaborative learning (CSCL) system designed to promote learning through group
activities.
In version 1, we
began by creating two simple applications:
BuddyGroup, which allowed for synchronous
communications between groups of users, and QuestionAnswering,
which allowed users to ask questions of a moderator, who could then respond
with an appropriate answer. In QuestionAnswering, all questions were parsed for important
keywords and were automatically scored using a machine learning algorithm to
assist the moderator in knowing which questions were most important. BuddyGroup also
provided a form for viewing performance statistics tracked for all members of
the user’s group. All messages passed
between users were automatically tracked and stored in a MySQL
database for archival purposes.
In version 2, we
added several tools for further utilizing the information stored in our
database. DatabaseSearch
was created to allow users to search through archived data and find previously
stored messages based on any combination of metadata stored about the
messages. It also allowed users to find
other users of the system, and learn how active they were. VirtualClassroom
collected statistics about every user and provided an overview of current
performance for moderators. Users with
problematic areas (as determined by thresholds set by the moderator) were
flagged for future observation. To
further analyze each statistic for every user, moderators could use StatCharts to view bar charts of every statistic for a
particular user, or one statistic for every user.
In version 3,
our goal was to combine all of our features into one capability. This served two purposes: 1) it made things easier for users, who now
had all functionality in one place, and 2) it made things easier for underlying
user and moderator agents because they could now access all available
information about user behavior and actions.
In previous versions, all intelligent functionality was contained in the
applications themselves, but 3.0 allowed us to merge all agent functionality
into one object. We also redesigned our
database for version 3, redoing the hierarchy of tracking. Before, all events recorded in CXP+I-MINDS
were associated with a specific session (i.e., a discrete instance of system
usage). In 3.0 and beyond, sessions were
now tied to specific classrooms, which allowed us to further separate data
between different classes using the same database. Finally, for 3.0, we also revised the CXP
networking stack (in CXP version 3.5) to use the Pragmatic General Multicast
(PGM) protocol to test reliability and latency in wireless environments.
In version 4, we
had several primary goals: 1) build a system robust enough to operate in both
wired and wireless environments, 2)
extend the existing functionalities to include 2 new modes – StructuredChat and QuizMode, 3)
reorganize the agent functionalities into specialties which are components that
specialize in one area of tasks, 4) create an XML configuration file to give
users more flexibility in setting options, 5) add network tracking for PGM and
wireless environments, and 6) redo the main CXP+I-MINDS form to offer more
functionality and be more user friendly.
All coding for 4.0 was moved to Visual Studio 2005 to use .NET 2.0 and
be compliant with CXP version 4.0.
Statistic tracking was also redone to provide less reliance on database
queries and faster performance, and the statistical analysis tools were redone
to offer better user interfaces and to take advantage of the new tracking
mechanisms. Finally, BuddyGroup
was renamed to BuddyChat to avoid confusion when
talking about the application and groups of buddies.
1.2. Wireless Issues and Logger Version
After a brief
deployment of version 3 in a wireless classroom environment, we quickly
discovered several flaws in our overall design.
One major flaw was our handling of database connections. In version 3, every object that needed
database access was given its own database connection for running queries. However, the database connections use TCP/IP
for their data transmissions, so in a classroom of over 30 students, where each
student has multiple connections, this resulted in too much concurrent TCP
traffic which slowed down the user’s connections due to loss and throughput
reductions and caused serious usability issues with 3.0. This was never a problem over wired
connections because they suffer from far less loss than wireless connections,
so slowdown was never previously an issue.
To counter this
problem, we built 4.0 with reduction in mind – how can we reduce the number of
database connections to ensure faster usability? How can we reduce the overall amount of
traffic being sent to protect against loss and need for retransmissions through
loss recovery? Reducing the impact of
CXP+I-MINDS on the network infrastructure it resides in allows our system to be
more robust and scalable, which are two important properties for a good
computer support collaborative learning (CSCL) application to have. To achieve our goal, we designed a new
paradigm for database access. First of
all, we eliminated all database connections from the software except for one
per user, which is shared by all of the objects within I-MINDS. We further added locks to protect against
multiple threads accessing the connection at once and implemented a stronger
error-prevention system using more try-catch handling of exceptions and error
logging. We also recognized that even 1
connection per user might be too many in wireless environments, so we designed
a logger system, where one instance of CXP+I-MINDS acts as a database mediator
for all participants. Whenever a user’s
software or agent needs to run a query, it simply sends a message to the
logger, which runs the query and returns the results. However, such a system introduces extra
messages into the system (especially because I-MINDS relies heavily on its
database to collect and store important information about users’ behavior and
actions), so we had to carefully design our message passing and refine our old
procedures to prevent the system from becoming congested with messages. Also, such a system is not necessary in wired
environments, so whether or not to use a local database connection can be set
in our new I-MINDS XML configuration file (described in section 1.5).
1.3. New Modes
To further
facilitate collaboration and learning amongst users utilizing CXP+I-MINDS, we
have introduced two new modes: 1) StructuredChat, and 2) QuizMode. StructuredChat is
just like BuddyChat, allowing students to carry on
chat-based discussions about multiple topics.
The main difference between the two modes is that BuddyChat
allows users to select their own groups, which StructuredChat
allows moderators to choose the groups for the students. StructuredChat is
described more in detail in Section 3.
QuizMode is primarily for use in educational or
training settings and allows moderators (teachers) to create questions and
quizzes to give to users (students) to assess their understanding of concepts
taught. Quizzes can either be given to
every student to be taken individually, or they can be given to structured
groups to work on collaboratively. QuizMode is described further in Section 4.
1.4. Agent Specialties
As
the functionalities and abilities of CXP+I-MINDS grow, so does the underlying
user and moderator agents. To facilitate future changes and maintain
current features, we have moved much of the agent functionalities out of its
main class and into specialties designed to work with specific tasks. For example, BuddyChat
and StructuredChat each have their own specialties
which contain all of the intelligence necessary for generating statistical
information about user behavior, notifying users of current conditions (such as
many unread messages in a topic or low contribution levels), and finding new
buddies based on topic discussions (in BuddyChat
only). A modeling specialty was also
created to handle all modeling tasks, such as tracking statistical information
from the various modes and for tracking the current network conditions (if
applicable). This allows us to build
statistical information incrementally, instead of relying on database queries
to generate values on demand (as was done in previous versions of CXP+I-MINDS),
greatly speeding up the analysis tools described in Section 5. Finally, a logger specialty was designed to
handle all database access for users and for archiving message content.
1.5. Configuration File
In previous
versions of CXP+I-MINDS, important information regarding the behavior of the
application was stored in various places.
For example, database connection information was stored in system DSNs in Windows.
However, during one deployment, we discovered how difficult this design
can be because users tend to forget to set the proper settings, and DSNs are not easy to find unless you are already familiar
with them. To make the system more
usable and prevent errors, we moved all configuration information into one XML
file which is read upon starting CXP+I-MINDS.
Within this configuration file is the database connection (with
encrypted password for security), whether or not we want to use our own
database connection or rely on a logger, and whether or not to enter debug mode
for extra error messages. For PGM within
the CXP networking stack, we also created a similar configuration system which
allows users to set all of PGM’s options (whether or
not to use late join, FEC, etc.) in one place.
This file also allows different settings for each socket using PGM. When we ported our PGM code to CXP 4.0, we
retained the UDP protocol, and the networking configuration file also allows
user to choose which protocol they would like to employ.
1.6. Network Tracking
One of our goals
as a research team is to provide an online collaboration system robust in
multiple networking environments. To
achieve this goal, we have been experimenting with the PGM networking protocol
in wireless environments. Unlike UDP
(which is often chosen for wireless networking), PGM provides reliability
through loss detection and recovery while maintaining high throughput (unlike
traditional TCP connections). In order
to get a better understanding of what is going on at the network level, we have
added network tracking to version 4 of I-MINDS.
Network tracking occurs at 3 levels: 1) the device level, 2) the socket
level, and 3) the application level. At
the device level, if users are using a wireless card installed with a special
NDIS protocol driver (which we can provide to users), the modeling specialty
can gather information about the network connection including which network and
access point (AP) the user is connected to, the signal strength, and the number
of available APs.
At the socket level, if the users are using PGM, we can get valuable
statistical information from the operating system, including how many packets
have been lost, how many have been recovered, what the current transmission
rate is, etc. These statistics are
available from both the sender and receiver perspective. Finally, at the application level, the user
agents occasionally pass special messages between one another to determine the
current reliability and latency in the network.
Right now,
network tracking is only performed for the purposes of our research for testing
network stability. However, we are
working on adding functionality to the agents to allow them to use information
about the current network conditions to adjust their transmission strategies to
alleviate pressure on the network and improve overall performance.
1.7. Main Form
To make
CXP+I-MINDS easier to use while providing increased capabilities, we redesigned
the main form to provide easy access to each mode and tool. We have also improved the functionality for
joining and leaving classrooms. Finally,
we have added an online user list to show users and moderators who are
currently in the same classroom and which modes they are using. This same list also allows participants to
invite others to use the different modes.
For users, it shows whether or not online participants are in their
buddy or structured groups, as shown in Figure 1.1.
Figure 1.1: Main IMINDS
Form for User Version
|
|
QuestionAnswering |
As with previous
versions, the question answering mode in version 4 retains both the user and
moderator version. Both versions were
originally based on the chat capability that comes with ConferenceXP. In version 3, the moderator version was
heavily modified to enable lists of previously received questions to be
displayed and sorted. Based on feedback
from a test run by
|
|
BuddyChat and StructuredChat |
In version 2,
buddy group was its own capability modified from the chat capability included
with ConferenceXP, which allowed users to choose a
group of buddies that they wished to converse with. In CXP+I-MINDS version 3, buddy group became
a capability mode of merged capability.
Along with the additions of advanced chat functionality it included a
buddy manager from which students could alter who is and who is not included in
their buddy group. In version 4, the
options menu from which to access ‘Display Statistics’, ‘Database Search’ and
‘Manage Buddy’ were moved to the main CXP+I-MINDS form. Also, with the addition of moderator assigned
structured groups, a new communication mode was added named Structured Chat.
The functionality is nearly identical, with the exceptions that users cannot
change their structured group members, and if their structured group is changed
by the administrator, the user will receive the current topics being discussed in
the new group they enter. Because of
their marked similarities, we will only describe the functionality of
Structured Group Chat, although the description applies to both.
|
|
QuizMode |
Another new mode
introduced in version 4 of CXP+I-MINDS is QuizMode. This mode is especially useful in educational
settings and allows moderators to create quizzes for users to take. All quiz content is stored in our database
for automatic retrieval when starting the mode, and all quiz results are
persistently stored as well for future analysis using the QuizStats
tool (described in Section 5.4). For
each quiz, moderators can also include support material in the form of files
for users to open that might provide more information or help on particular
questions. Support material are
automatically saved in the CXP+IMINDS directory on both moderator and user
machines to protect against accidental deletion of the original file and to insure
that the files are available until the moderator no longer wants them. Please note that due to ConferenceXP
transmission limitations, total support material for a particular quiz cannot
be larger than 750KB in size.
|
|
Analysis Tools |
To assist both
users and moderators analyze the data tracked by CXP+I-MINDS, we have designed
several tools: 1) VirtualClassroom,
2) Statistics, 3) DatabaseSearch, and 4) QuizStats. All four
of these tools are available in the Moderator version, while users only have
Statistics and DatabaseSearch. QuizStats
is the only new analysis tool in version 4 of CXP+I-MINDS, but we have modified
VirtualClassroom to provide more statistical
information based on the increase in statistical tracking done in version
4. In versions 2 and 3, VirtualClassroom had a limited set of statistics, and also
included an overview of all message traffic from the system in the form of
message and topic lists, but we removed these because they were not used during
deployment. Also, for version 4, we
included the Statistics tool in the user version to replace its text-based
predecessor, which was available in all prior versions of CXP+I-MINDS, to accommodate
the increase in statistical tracking and to provide information in an easier to
analyze format. DatabaseSearch
has remained unchanged from its design in version 3.
All statistical
modeling is done by each user agent based on their user’s interactions with
other users. Moderators receive their
statistics from the user agents and the values stored in the database. For version 4, we added new statistics, and
grouped the statistics by mode instead of by type as was done in previous
versions. The current set of statistics
used by the Virtual Classroom and Statistics tools are shown in the tables
below:
|
QuestionAnswering |
BuddyChat/StructuredChat |
|
Questions Asked Per Session |
Messages Sent Per Session |
|
Questions Answered Per Session |
Number of Topics Started Per Session |
|
Questions Broadcast Per Session |
Number of Topics Started by Others Per Session |
|
Average Question Length |
Responses Received Per Session |
|
Average Keywords in Questions |
Average Dialog in Topics Started |
|
Average Question Score |
Average Self Dialog Improvement |
|
Max Question Score |
Average Other Dialog Improvement |
|
Min Question Score |
Average Response Time |
QuizMode |
|
Number of Quizzes |
|
Percent of Quizzes Completed on Time |
|
Average Quiz Duration |
|
Average Questions Per Quiz |
|
Percent of Questions Seen |
|
Percent of Questions Correct |
|
Percent of Questions Incorrect |
|
Percent of Questions Unanswered |
|
Percent of Points Earned |
|
Average Token Requests Per Quiz* |
|
Average Tokens Granted Per Quiz* |
|
Questions Seen With Token Per Quiz* |
|
Questions Answered With Token Per Quiz* |
|
Average Time With Token Per Quiz* |
* Collaborative Quiz only
Tables 5.1 - 5.3: Statistics for each Mode