About
GrooveNet is a research-grade system that explores real-time, music-driven dance movement generation using artificial neural networks. The project, a collaboration between Omid Alemi and Jules Françoise at Simon Fraser University's School of Interactive Arts and Technology, aims to teach a neural network to produce beat-synchronous and musically coherent dance motions for any given song. Rather than relying on supervised learning, GrooveNet treats the problem as unsupervised learning. A custom database of synchronized groove moves and songs was created as training data. For each song, 84-dimensional audio descriptors are extracted—covering low-level features (RMS, Bark bands), spectral features (centroid, flux, rolloff, etc.), timbral features (MFCCs, Tristimulus), and melodic features (pitch, inharmonicity, dissonance). These descriptors are fed alongside joint rotation data into a Factored Conditional Restricted Boltzmann Machine (FCRBM). The model learns movement patterns from labeled mocap segments and can then generate new dance sequences conditioned on audio from both training songs and previously unheard tracks. Results were presented at the Workshop on Machine Learning for Creativity at ACM SIGKDD 2017. GrooveNet is primarily aimed at AI researchers, creative technologists, and developers interested in the intersection of music, motion, and machine learning. It represents a compelling proof-of-concept for generative AI applied to embodied, time-synchronized creative expression.
Key Features
- Beat-Synchronous Dance Generation: Generates dance movements that are temporally aligned with the beat and rhythm of a given song using an FCRBM architecture.
- 84-Dimensional Audio Feature Extraction: Extracts rich audio descriptors including spectral, timbral, and melodic features to condition movement generation on musical content.
- Generalization to Unheard Songs: The model can generate plausible dance sequences for songs it was never trained on, demonstrating musical generalization.
- Unsupervised Multi-Modal Learning: Jointly trains on audio descriptors and joint rotation data without requiring manually labeled movement-music pairs.
- Custom Synchronized Mocap Database: Built on a purpose-created dataset of synchronized groove movements and songs recorded at multiple takes for robust training.
Use Cases
- Academic research into AI-generated choreography and music-motion synchronization
- Prototyping generative dance animations synchronized to music for games or virtual environments
- Exploring unsupervised multi-modal learning with audio and motion capture data
- Generating rhythmically coherent avatar animations for music visualization applications
- Studying how neural networks can learn latent movement patterns from unlabeled mocap sequences
Pros
- Novel Research Approach: Frames music-to-dance as an unsupervised problem, enabling the system to discover movement patterns without hand-labeled supervision.
- Generalizes Beyond Training Data: Successfully generates dance movements for songs not seen during training, showing real-world applicability.
- Rich Audio Conditioning: Uses a comprehensive 84-dimensional feature set covering spectral, timbral, and melodic properties for musically aware generation.
Cons
- Research Prototype Only: GrooveNet is an academic project without a production-ready interface, API, or easy-to-use deployment pipeline.
- Limited Dataset Size: The custom mocap database is relatively small and constrained to groove-style movements, which may limit diversity of outputs.
- Development Appears Stalled: The project dates to 2017 and does not appear to have received public updates since, limiting its relevance to current state-of-the-art.
Frequently Asked Questions
What is GrooveNet?
GrooveNet is a research project that uses a multi-modal neural network (FCRBM) to generate beat-synchronous dance movements conditioned on audio features extracted from a given song.
How does GrooveNet generate dance movements?
It extracts 84-dimensional audio descriptors from music and trains an FCRBM on both these descriptors and motion capture joint rotations, learning to produce synchronized dance sequences at inference time.
Can GrooveNet work with any song?
Yes. The model has been tested on songs not seen during training and can generalize its dance generation to unheard music by conditioning on extracted audio features.
Is GrooveNet available to use or download?
GrooveNet is an academic research project. Demo outputs and a published paper (KDD 2017 workshop) are available on the project page, but there is no public software release or API.
Who created GrooveNet?
GrooveNet was created by Omid Alemi and Jules Françoise, researchers at the School of Interactive Arts and Technology at Simon Fraser University, Canada.
