Seeing Who Is Signing and With Which Hand

Abstract

This study is a computer vision analysis of 4.5 hours of video data from 40 signers in the Swedish Sign Language Corpus, aiming to evaluate the reliability of classifying 1) who the main signer is at any given time during dyadic conversation, and 2) the dominant hand (i.e., handedness) of each signer. First, the distance moved by the hands of each signer is used to compare the manual activity between a) the two signers to determine whose hands are more active, and b) the hands of each signer to determine which hand is more likely to be dominant. Second, the height of the hands is used to compare their prominence in signing space between a) the two signers to determine whose hands are more prominent, and b) the hands of each signer to determine which hand is more likely to be dominant. The results show that while both distance and height approaches can reliably classify – individually or combined – the main signer in any segment of a conversation, the height approach is better at determining the overall handedness (right- or left-dominant) of signers. For the handedness classification, the optimal method turns out to be a two-step approach, first classifying the main signer per segment, then using only signer-relevant segments to classify handedness.

Resources

Details