SmartWatch for Wall Writing: Real-time Transcription of Wall Writing from Inertial Sensing

This paper presents a smartwatch-based application, called WatchScribe, to convert any wall writing to virtual board-works. Users can scribe anything over this virtual whiteboard with zero-bootstrapping by mimicking the writing on the wall with a pen. Transcribing wall-writing with a smartwatch is challenging because of two reasons. (a) As the smartwatch's orientation changes continuously due to wrist movements, the pen's locus is different from that of the smartwatch. (b) There are events when the user lifts the pen from the whiteboard and starts at a different position to draw the next stroke. WatchScribe leverages locomotive data from a smartwatch to estimate the pen's locus using an unsupervised method. Further, it uses a lightweight approach to extract the writing micro-gestures from the estimated pen's locus and reproduces a real-time transcription of the wall-writing. We apply qualitative human-driven and novel quantitative metric-based evaluation techniques for checking the correctness of WatchScribe over the data collected from 10 participants in a very diverse setup. Compared to the closest baseline that uses a smartphone's corner as a pen, we achieve an average 9% reduction in disparity score and 27% improvement in average accuracy in terms of decipherability of the reproduced transcripts.