SignLLM - A Multilingual Sign Language Generation Model that Generates Corresponding Sign Language Videos from Text Input

What is SignLLM

SignLLM is an innovative multilingual sign language generation model that generates corresponding sign language videos from text input. It is the world's first model to support multiple countries' sign languages, covering eight languages including American Sign Language (ASL), German Sign Language (GSL), Argentine Sign Language (LSA), and Korean Sign Language (KSL). The model was developed based on the Prompt2Sign dataset, automatically collecting and processing sign language videos on the internet through automation technology, and combining a new loss function and reinforcement learning module to achieve efficient data extraction and model training.

SignLLM's Main Features

• Sign Language Video Generation: Converts input text into natural and fluent sign language gesture videos, suitable for multiple languages.
• Multilingual Support: Supports eight sign languages, covering different countries and regions.
• Efficient Training and Optimization: Accelerates the training process through a reinforcement learning module and improves data sampling quality.
• Style Transfer and Fine-Tuning: Converts the generated model output into realistic sign language videos, approaching a real human appearance.
• Education and Translation Support: Can be used for sign language teaching, sign language translation, and providing communication support for the deaf community.

SignLLM's Technical Principles

• Discretization and Hierarchical Representation: SignLLM achieves discretization and hierarchical representation of sign language videos through two key modules. First, the Vector Quantized Visual Sign (VQ-Sign) module decomposes sign language videos into a series of discrete character-level tokens, similar to characters in language. Then, the Codebook Reconstruction and Alignment (CRA) module combines these character-level tokens into word-level tokens, forming hierarchically structured sign language sentences.
• Self-Supervised Learning and Context Prediction: The VQ-Sign module performs self-supervised learning through context prediction tasks, rather than traditional video reconstruction methods. It can capture the temporal dependencies and semantic relationships of sign language videos without reconstructing high-dimensional video data.
• Sign-Text Alignment: To further improve the semantic compatibility between sign language tokens and text tokens, SignLLM uses the Maximum Mean Discrepancy (MMD) loss function to align the embedding space of sign language tokens with the embedding space of text tokens.
• Combination with LLM: SignLLM combines the generated sign language sentences with a frozen LLM, guiding the LLM through text prompts to generate translations in the target language. This enables SignLLM to achieve efficient sign language-to-text translation based on the powerful translation capabilities of LLMs.
• Training and Inference: SignLLM's training is divided into two stages: pretraining and fine-tuning. The pretraining stage includes context prediction tasks and codebook alignment, while the fine-tuning stage further optimizes model performance.

SignLLM's Project Address

• Project Official Website：https://signllm.github.io/
• Github Repository：https://github.com/SignLLM
• arXiv Technical Paper：https://arxiv.org/pdf/2405.10718

SignLLM's Application Scenarios

• Education Field: SignLLM can serve as a virtual sign language teacher, converting text into sign language gesture videos, helping students learn sign language more intuitively and accelerating the learning process.
• Medical Scenarios: In medical environments such as hospitals, SignLLM can convert doctors' speech or text into sign language in real time, helping hearing-impaired patients describe symptoms more accurately and understand medical advice, improving the medical experience.
• Law and Public Services: In court or legal consultations, SignLLM can provide accurate sign language translation, ensuring that hearing-impaired individuals have fair communication opportunities in legal matters. In addition, in public services or customer service, it can provide instant sign language translation, making it convenient for the hearing-impaired community.
• Entertainment and Media: SignLLM can provide real-time sign language translation for movies, TV programs, or online videos, enriching the cultural life of the hearing-impaired community.
• Daily Life: Individual users can use SignLLM for daily communication, such as chatting with hearing-impaired friends or communicating in quiet environments.