With the autopilot off, the trim wheel should only move if you command trim changes, either by clicking on the wheel itself or using the elevator trim control assignments. Do you have a control axis assigned to the elevator trim that might be moving by small amounts without any input by you?