The data science aspects of this position involve monitoring cross validation of scikit-learn SVC and KPI models daily, and DNN models from data collected by the peer learning application weekly. The KPIs include:
A. Validity. The baseline is the ratio of accuracy variance to instructor inter-rater agreement variance, relative to “System-Human agreement,” e.g. 58.4% (Chen et al., ETS, 2018.) Validity components include:
1. The number of speaking exercises assigned and performed;
2. The number of listening-typing exercises assigned and performed;
3. The number of minutes each type of exercise has been performed;
4. Intelligibility-assessable words and phrases, for each language;
5. Number of branching scenario interactions, for each language;
6. Spoken recordings in total, and per assessable words and phrases;
7. Transcripts collected in total, and per spoken recordings;
8. Exemplar pronunciation recordings identified per assessable prompts;
9. Spoken remediation responses provided; and
10. Observed intelligibility difference each student has achieved on their assigned groups of words and phrases.
B. Ease of use. Components include:
(1) the median duration required to complete assignments achieving a specific level of validity,
(2) the proportion of assignments completed, and
(3) the median numbers of (a) button-presses and (b) utterances required to complete assignments.
C. Anomalies, including error log analysis and resolution, system resource monitoring, ad performance, growth, scaling, etc.
