Correctness-prediction performance was measured using area under the receiver operating characteristic curve (AUC). The evaluation compared KQN and DKT + KQN with Bayesian Knowledge Tracing (BKT), Item Response Theory (IRT), Deep Knowledge Tracing (DKT), and Deep Key-Value Memory Networks (DKVMN). Overall, KQN achieved the best reported performance.

San Diego State University

Google

The Results and Analysis section of the Knowledge Query Network (KQN) paper evaluates the model along four dimensions: (1) Correctness Prediction, (2) Knowledge Interaction Visualization, (3) Skill Domain Analysis, and (4) Sensitivity Analysis of the Vector Space Dimensionality.

Results and Analysis (Knowledge Query Network for Knowledge Tracing)

In order to measure the impact of the embedding dimension on the models performance different values were tried, d = $d_{opt}$, $0.5d_{opt}$, 2$d_{opt}$.
For each pair from these three pairwise skill distance was calculated as:

$$\xi_{d_1, d_2}  = \frac {\sum_{i \gt j}|pdist_{d1}  (s_i, s_j) - pdist_{d_2}(s_i, s_j)|} {\begin{pmatrix} N\\ 2 \end {pmatrix}}$$
$$d_1 \neq d_2$$

$pdist_d(s_i, s_j)$ refers to the pairwise distance and N is number of skills.
Then the $\xi$ is compared to average pairwie distance $\mu$:

$$\mu_d =   \frac {\sum_{i \gt j}pdist_{d}  (s_i, s_j) } {\begin{pmatrix} N\\ 2 \end {pmatrix}}$$
According to this experiment, KQN learned the skill relationship in a better manner when d was higher. 
Additionally, the authors used Mantel tests to measure the similarity between 2 distance matrices. It showed that when predicting the probability of the correctness as well as for learning relation between skill vectors KQN model is stable for different d values.

Sensitivity Analysis and Vector Space Dimensionality (Knowledge Query Network for Knowledge Tracing)

In the KQN skill-domain analysis, Euclidean distance with average cluster linkage produced the highest Adjusted Rand Index (ARI) on the Synthetic-5 dataset. The authors then applied this selected configuration to the ASSISTments 2009–2010 dataset, varied the number of clusters, and evaluated accuracy using the area under the ROC curve (AUC).

Skill Domain Analysis (Knowledge Query Network for Knowledge Tracing)

Correctness Prediction (Knowledge Query Network for Knowledge Tracing)

The authors visualized skill-level correctness predictions for one student using the ASSISTments 2009–2010 dataset. As expected, correctly answering an item involving a skill generally increased the predicted probability of correctly answering another item involving that skill. However, the prediction sometimes also increased after an incorrect response, an unintuitive behavior also observed in Deep Knowledge Tracing (DKT).

Learn Before

Related