Send Face Tracking to VRChat

Send expression data from the capture app to VRChat via OSC

This page has been machine translated and may contain inaccuracies.
In this guide, the avatar display app of VRC Avatar Viewer is referred to as the "Viewer App", and the motion capture app (Motion Capture for VRC Avatar Viewer) is referred to as the "Capture App".

The capture app can send facial expression data estimated from a webcam feed to VRChat's OSC API. By integrating it with a VRCFaceTracking-compatible avatar, you can animate your VRChat avatar with rich expressions using just a single webcam.

Face tracking in VRChat
Face tracking in VRChat

What you need

  • Motion Capture for VRC Avatar Viewer (the capture app / distributed on BOOTH)
  • Webcam (720p or higher recommended)
  • VRChat (Steam / standalone version)
  • A VRCFaceTracking-compatible avatar
[Webcam] → [Capture App] →OSC / UDP :9000→ [VRChat]
Tips
  • This feature works independently of the VRC Avatar Viewer. If you only want to animate facial expressions in VRChat, you don't need to launch the viewer
  • You can also enable it simultaneously with VMC transmission to the VRC Avatar Viewer. The same expressions can be played back in both VRChat and the viewer
1

VRChat-side setup

1-1. Enable OSC

VRChat has OSC reception disabled by default. Enable it from the in-game menu.

  1. Launch VRChat and enter a world
  2. Open the Quick menu and go to Options → OSC
  3. Turn "Enabled" ON
Tips
  • The default OSC reception port is 9000. The capture app uses this port as well
  • If you've previously used another OSC app, an old parameter configuration file may remain. If expressions don't work as expected, use Reset Config to regenerate the configuration file (in VRChat's OSC menu)

1-2. Use a VRCFaceTracking-compatible avatar

Your VRChat avatar must have VRCFaceTracking-compatible facial expression parameters built in. With unsupported avatars, nothing will happen even if you send data from the capture app.

Tips
  • If you're not sure whether your avatar is supported, check the avatar's distribution page for mentions of "VRCFaceTracking-compatible" or "VRCFT-compatible"
  • If you want to add support to an unsupported avatar, many face tracking add-ons for various avatars are available on BOOTH
2

Capture app settings

2-1. Start the camera

  1. Launch the capture app
  2. Select "Camera", "Resolution", and "FPS", then press "Start Camera"
  3. Once you appear in the preview, stand in front of the camera and stay still for 1-2 seconds (initial calibration)
Capture app screen
Capture app screen
Tips
  • If you want to play VRChat with an HMD while using this, place the webcam below or in front of your face
  • Sufficient lighting to clearly show your face is required. The accuracy of expression estimation drops significantly in dim conditions

2-2. Turn on VRChat OSC transmission

  1. Confirm that "VRC OSC Port" in the capture app is set to 9000
  2. Press the "VRChat OSC" button to turn it ON
Capture app screen
Capture app screen

When the button is turned ON, the capture app will start sending face tracking values to 127.0.0.1:9000 in the format /avatar/parameters/v2/<parameter name>.

Tips
  • The transmission rate is automatically limited to about 30fps. This is to reduce network load and processing load on VRChat, and is sufficient for tracking expressions
  • The capture app and VRChat are assumed to run on the same PC. Configurations that send data from a separate PC are not supported
3

Verify operation

Success is when your avatar's expressions in VRChat move in sync with the webcam feed. Here are some good points to test.

  • Blinking Open and close both eyes to check if the avatar's blinking responds
  • Mouth movement Move your mouth saying "A, I, U, E, O" to check if lip sync works
  • Eyebrows Raise and lower your eyebrows to check if it's reflected on the avatar
  • Gaze Move your eyes left/right and up/down to check if the avatar's eyes follow
Face tracking in VRChat
Face tracking in VRChat
Tips
  • Some parameters (cheeks, tongue, etc.) cannot be tracked with a typical webcam. Even if implemented in the avatar, some may not move
  • If you feel expression movements are weak, you can adjust responsiveness and motion amount in the capture app's "Motion Adjustment" section