Whisper Misreads Hums and Gets Timing Wrong with the Wrong Stem

I was transcribing a 280.85-second experimental song to get vocal phrase timings for edit planning. On the first pass, I transcribed the full song mix and got a file that looked complete, with start and end times for every segment. Before using it, I tried another pass on the vocal stem separated with Demucs and compared the two runs. Several points did not match at all, and the differences were more than minor. In some places, the text did not exist in the actual audio.

This article covers only the traps in the "results"; the GPU workflow and delivery with hash verification are covered in Demucs + Whisper, One File at a Time on a GPU Machine, Then Retrieve Results and Verify SHA-256. I will not quote a single line of text transcribed by Whisper. What I need here is the timing and the patterns of failure.

Overview: Nearly One-Fifth of the Mix Transcript Was Hallucinated

I will start with the numbers I actually measured.

I calculated "about 20%" myself from two segments (27.5 and 30.0 seconds, totaling 57.5 seconds, divided by 280.85). The system did not report that number, so I want to label it as a calculation.

Trap 1: Instrumental Passages Become Tens of Seconds of Musical Notes

The mix transcription contained segments at 96.72-124.24 seconds (27.5 seconds) and 246.52-276.50 seconds (30.0 seconds) whose text was a long string of repeated musical-note symbols. Those passages were purely instrumental, with no vocals. Whisper was designed to transcribe speech, and when it encountered audio without words, it produced this string of symbols (any deeper explanation is speculation; I have not proved it). These segments had neat start and end times, just like real ones. Using them as "sections with vocals" would produce more than a minute of error in total.

Trap 2: Humming Is Read as Sentences

The opening passage at 3-19 seconds was actual humming, with no words, but Whisper read it as Thai sentences that were not there. I confirmed that it was humming from the vocal stem and pitch analysis, rather than from the transcribed text. The text appeared just as confident as real speech. This is the most dangerous point, because it is wrong without any warning.

I used pitch tracking on the vocal stem to confirm it. If a passage has a continuous pitch but unreliable text, it is likely to contain wordless vocals.

# hum_check.py — ช่วงนี้มีระดับเสียงต่อเนื่องไหม (ใช้ librosa.pyin บนสเต็มเสียงร้อง)
import numpy as np, librosa

def voiced_ratio(vocals_wav, t0, t1):
    y, sr = librosa.load(vocals_wav, sr=22050, offset=t0, duration=t1 - t0)
    f0, voiced_flag, voiced_prob = librosa.pyin(
        y, fmin=librosa.note_to_hz('C2'), fmax=librosa.note_to_hz('C6'), sr=sr)
    return float(np.mean(voiced_flag))          # สัดส่วนเฟรมที่มีระดับเสียงชัดเจน

r = voiced_ratio('sep/vocals.wav', 3.0, 19.0)
print(f'voiced {r:.0%} ของ 3-19 s')             # สูง = มีเสียงร้อง แม้ข้อความจะน่าสงสัย

Trap 3: A Nonexistent Greeting at the End of the File

The vocal-stem transcription contained a segment at 279.44-280.85 seconds whose text was a short greeting, even though that passage was almost silent. This is a common Whisper hallucination with silence at the end of a file. I will not reproduce that sentence here.

A related symptom was a long segment at 264.84-277.92 seconds (13 seconds) with very short text repeated over and over, indicating a repetition loop. A long segment with little text is always suspicious.

Filters I Propose (Not the Settings I Actually Used)

Whisper already provides confidence-related fields in every segment: avg_logprob, no_speech_prob, and compression_ratio. As I recall, the openai-whisper defaults are compression_ratio_threshold 2.4, logprob_threshold -1.0, and no_speech_threshold 0.6. This is general knowledge; check the documentation for the installed version before citing it. The filters below are proposals written for readability. I did not use these exact settings in the project, and they need tuning against your own files.

# filter_segments.py — ข้อเสนอ: ติดธงเซกเมนต์ที่น่าสงสัย ไม่ลบทิ้งเงียบ ๆ
import json, sys, unicodedata

def has_letters(text):
    return any(unicodedata.category(c)[0] in 'LN' for c in text)    # ตัวอักษรหรือตัวเลขจริง

def reasons(seg):
    r = []
    if not has_letters(seg['text']):          r.append('symbols-only')
    if seg['compression_ratio'] > 2.4:        r.append('repetitive')
    if seg['avg_logprob'] < -1.0:             r.append('low-confidence')
    if seg['no_speech_prob'] > 0.6:           r.append('probably-silence')
    if seg['end'] - seg['start'] > 10 and len(seg['text'].strip()) < 12:
        r.append('long-span-short-text')      # เกณฑ์ที่ผมเพิ่มเอง ใช้จับลูปซ้ำ
    return r

segs = json.load(open(sys.argv[1], encoding='utf-8'))
for s in segs:
    s['flags'] = reasons(s)
    if s['flags']:
        print(f"[WARN] {s['start']:.2f}-{s['end']:.2f}  {','.join(s['flags'])}")
json.dump(segs, open(sys.argv[2], 'w', encoding='utf-8'), ensure_ascii=False, indent=1)

I chose to flag segments instead of deleting them because automatic filters can make errors in either direction. If a segment is silently deleted, nobody will know it was ever there.

Trap 4: Group Vocals Are in the Wrong Stem

One group-vocal section at 62.4-83.5 seconds (about 21 seconds) was absent from the lead-vocal stem and was in the backing-vocal stem instead. Transcribing only the lead-vocal stem would give no timing for that section at all. The backing-vocal stem needed a separate transcription pass.

This means the transcription step needs to run through every stem containing human vocals, rather than choosing one stem and assuming it is complete. In my set, those were the lead vocals and backing vocals. The instruments did not need transcription.

Trap 5: The First Vocal Section Starts 0.7 Seconds Off

From the mix, I believed the first vocal section started at 19.2 seconds. The lead-vocal stem gave 18.5 seconds, which was 0.7 seconds earlier. That is about 21 frames at 30 fps, clearly visible in a cut timed to the beat. If I trusted the mix timing, a cut placed there would be 0.7 seconds late without any warning.

Trap 6: Segment Boundaries Differ in Dense Audio

Comparing the two runs (mix and stem transcriptions), clear passages matched closely, differing by only 0.02-0.1 seconds, such as 124.20 versus 124.24 or 127.50 versus 127.48. Dense passages, however, were segmented differently. One clear example was the stem grouping 158.54-166.42 seconds into one segment, while the mix split it into two (158.46-161.14 and 162.78-166.44). I therefore need to compare two sources before trusting segment boundaries.

# compare_runs.py — เทียบเวลาเริ่มของสองรอบ (มิกซ์ vs สเต็ม)
import json, sys

a = json.load(open(sys.argv[1], encoding='utf-8'))      # รอบถอดจากมิกซ์
b = json.load(open(sys.argv[2], encoding='utf-8'))      # รอบถอดจากสเต็ม
b_starts = [s['start'] for s in b]

for s in a:
    near = min(b_starts, key=lambda t: abs(t - s['start']))
    d = near - s['start']
    mark = 'OK ' if abs(d) <= 0.1 else 'DIFF'
    print(f'[{mark}] a={s["start"]:7.2f}  b={near:7.2f}  ต่าง {d:+.2f} s')

Lines marked DIFF are places I need to listen to myself, rather than places where I should automatically choose one side.

Trap 7: Phrases ASR Cannot Transcribe Must Be Marked as Estimates

There were passages where ASR could not transcribe the words, but pitch confirmed that vocals were present. One example was 53.7-61.3 seconds, spanning two lines. I divided the time equally and made the estimated flag mandatory in the data so nobody would mistake those numbers for actual measurements.

# split_estimated.py — แบ่งช่วงที่ถอดไม่ได้เท่า ๆ กัน พร้อมธงบังคับ
def split_equal(t0, t1, n):
    step = (t1 - t0) / n
    return [{'start': round(t0 + i * step, 3),
             'end':   round(t0 + (i + 1) * step, 3),
             'timing': 'estimated'}               # ห้ามขาด ธงนี้คือความซื่อตรงของข้อมูล
            for i in range(n)]

# ตัวอย่างการคำนวณ: ถ้าส่วนที่ถอดไม่ได้คือ 53.7-60.0 s (คำที่จับได้เริ่มที่ 60.0 s) มี 2 บรรทัด
for seg in split_equal(53.7, 60.0, 2):
    print(seg)                                    # 53.7-56.85 และ 56.85-60.0

The example above is an illustrative calculation, rather than a measurement. Dividing the time equally is the most honest guess when there is no data, but it must be labeled as a guess.

Read the Transcript Before Planning Edits: My Check Order

Before using transcript timings to place cuts, I check in this order every time. First, I look at the overall picture and count unusually long segments, because segments lasting tens of seconds in a song with intermittent vocals are almost certainly instrumental passages read as text. Next, I check the end and beginning of the file, where the audio is quietest and Whisper often fills things in. Third, I compare against a transcription from another source. If the two runs agree within a few hundred milliseconds, I can have some confidence; if they disagree, I listen to that passage.

I choose flags over deletion because automatic filters can make errors in either direction. An overly strict filter will discard real segments with quiet vocals. An overly loose filter will let hallucinations through. With flags in the data, later readers can see both what was retained and why it was considered suspicious, and make their own decisions. As for the thresholds I proposed above, anyone using them needs to try them on their own files first. I did not use these exact settings in the project.

Another thing I learned from this work is that model confidence is not accuracy. A field such as avg_logprob tells me how confident the model is in what it typed, rather than whether that content actually exists in the audio. Humming read as sentences is the clearest example. That is why I always need evidence from another channel. In this case, it was pitch measured from the signal itself.

Practical Takeaways

1. Do not transcribe only the full song mix. Transcribe the separated vocal stems, and cover every stem containing human vocals.
2. Flag segments that contain only symbols, are long with little text, have low confidence, or are likely to be silence, then listen to them yourself.
3. Confirm humming with pitch, rather than the transcribed text.
4. Compare two runs before trusting segment boundaries. Clear passages agree within 0.1 seconds; listen to dense passages.
5. Divide untranscribable passages into equal times, with a mandatory estimated flag.
6. Set human_listening_verified to false until a person actually listens.

The shortest lesson is that a transcript with neat timestamps does not prove that anything is present in the audio at those times. I always need another layer of evidence, such as pitch or the correct stem, and uncertain data must carry flags in the file itself, rather than relying on memory. If I have left anything unexplained or made a typo, I will come back and fix it later.