MS-G3D
Real-Time Single Hand Gesture Recognition
RUN
MS-G3D is a real-time hand gesture recognition tool powered by the Multi-Scale Graph 3D Convolutional Network architecture, running entirely in your browser with no data leaving your device. It uses MediaPipe Hands to extract 21 3D landmarks per hand, then feeds a temporal sequence of those landmarks through the MS-G3D pipeline implemented in TensorFlow.js.
The MS-G3D architecture (Liu et al., CVPR 2020) combines two graph convolution branches:
- Multi-Scale Graph Convolution (MS-GCN): aggregates spatial features from k-hop neighbourhoods on the hand skeleton graph (scales k = 1, 2, 3), letting the network capture both local finger articulations and global hand shape simultaneously.
- Spatio-Temporal G3D: unfolds a sliding temporal window across the skeleton sequence and applies a joint spatial–temporal convolution, capturing how joints move in relation to each other over time.
The tool ships with a geometry-based classifier that works immediately without any training. A built-in custom gesture training mode lets you record your own gesture samples and fine-tune the classification head in-browser (≈2 seconds on CPU).
The following 8 gestures are recognised out of the box:
- Fist ✊
- Open Hand ✋
- Point ☝️
- Peace ✌️
- Thumbs Up 👍
- Rock 🤘
- Call Me 🤙
- OK 👌
The data is exported as a CSV file structured as follows:
| frame | timestamp | tag | hand | gesture | confidence | joint_0_x | joint_0_y | joint_0_z | … | joint_20_x | joint_20_y | joint_20_z |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 42 | 1718000000 | baseline | Right | Peace | 0.913 | 0.512 | 0.748 | -0.031 | … | 0.421 | 0.302 | -0.089 |
Landmark coordinates are in MediaPipe's normalised image space (x, y ∈ [0, 1]). The z value is depth relative to the wrist (negative = closer to camera). Rows where no hand is detected are omitted.
Reference: Liu Z., Zhang H., Chen Z., Wang Z., Ouyang W. Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition. CVPR 2020. arXiv:2003.14111