Video Processing Pipeline — From Raw Footage to Polished Content

---

# 🎬 Video Processing Pipeline

## From Raw Footage to Polished Content ✨

@luk036 👨‍💻 · 2026 📅

---

### 📋 Agenda

### 🛠️ Skills Created

1. **SRT Subtitle Fixing** — `fix-srt` / `validate-srt` 📝
2. **Silent Gap Removal** — `remove-silent-gaps` 🔇
3. **Chinese Filler Removal** — `remove-filler-zh` 🗣️

### 💡 Key Challenges

4. Windows Temp File Bug 🪟🐛
5. Timeout Estimation ⏱️
6. Word-Level vs Segment-Level 🎯

]

### 📊 Results

7. Full Pipeline Demo 🚀
8. Before & After 📈
9. Key Takeaways 🏁

]

---

## 🛠️ Skill 1: SRT Subtitle Fixing

---

### 📝 `fix-srt` — Intelligent Transcription Repair

**What it does:**
- Removes Chinese filler words from SRT files (呃, 啊, 哦, 哈, 唉)
- Fixes **70+** common ASR transcription errors
- Validates subtitle structure after fixing

**Example fixes:**
| Original | Corrected |
|----------|-----------|
| `deep CV four` | DeepSeek V4 |
| `offer lap` | overlap |
| `封神` | 函数 (function) |
| `headlock` | deadlock |

]

**Transcription Error Categories:**

<pre>
graph TD
 ASR[ASR Output] --> TE[Transcription Errors]
 TE --> CP[CS Terms e.g. sorting, overlap]
 TE --> TL[Tools e.g. DeepSeek, OpenCode]
 TE --> CN[Chinese Homophones e.g. 封神 → 函数]
 TE --> FG[Filler Words e.g. 呃, 啊, 哦]
 style ASR fill:#5e81ac,color:#eceff4
 style CP fill:#bf616a,color:#eceff4
 style TL fill:#d08770,color:#eceff4
 style CN fill:#b48ead,color:#eceff4
 style FG fill:#a3be8c,color:#2e3440
</pre>

]

---

### 🔍 `validate-srt` — Subtitle Quality Assurance

**Structural checks performed:**

| Check | What it catches |
|-------|----------------|
| Numbering | Missing or duplicated sequence numbers |
| Timestamp format | Malformed `-->` lines |
| Chronological order | `start >= end` or overlapping entries |
| Content quality | Empty text, suspiciously short entries |

**Example output:**

```
PASS: subtitles.srt
验证通过：共计 292 条字幕

FAIL: broken.srt
3 error(s):
#2: expected number 2, got 4
#3: start >= end (00:00:03,000 --> 00:00:01,500)
#5: starts before previous entry ends (gap: -500ms)
```

---

## 🔇 Skill 2: Silent Gap Removal

---

### `remove-silent-gaps` — Pipeline Overview

<pre>
graph LR
 IN["🎬 input.mp4 1116s"] --> SP[Split -c copy]
 SP --> C1["Chunk 1 372s"]
 SP --> C2["Chunk 2 372s"]
 SP --> C3["Chunk 3 372s"]
 C1 --> DA1["🔇 remove-dead-air (FFmpeg silencedetect)"]
 C2 --> DA2["🔇 remove-dead-air"]
 C3 --> DA3["🔇 remove-dead-air"]
 DA1 --> T1["Trimmed 1 330s"]
 DA2 --> T2["Trimmed 2 307s"]
 DA3 --> T3["Trimmed 3 265s"]
 T1 --> CN["Concat -c copy"]
 T2 --> CN
 T3 --> CN
 CN --> OUT["🎬 output.mp4 902s (-19%)"]
 style IN fill:#bf616a,color:#eceff4
 style OUT fill:#a3be8c,color:#2e3440
 style SP fill:#5e81ac,color:#eceff4
 style CN fill:#5e81ac,color:#eceff4
 style DA1 fill:#b48ead,color:#eceff4
 style DA2 fill:#b48ead,color:#eceff4
 style DA3 fill:#b48ead,color:#eceff4
</pre>

]

---

### 🔧 FFmpeg Silence Detection

**How it works:**

1. `silencedetect` filter scans audio for sections below a threshold
2. Nearby silences are **merged** (default gap: 0.3s)
3. Keep segments are computed by **inverting** silence ranges
4. FFmpeg `select`/`aselect` filters cut the video at precise boundaries

```
Detecting silence (threshold=-40dB, min_duration=0.5s)...
  Found 54 silence segments
  Merged into 53 segments (gap=0.3s)
  Dead air to remove: 42.4s (11.4% of chunk)
  Keep segments: 54
```

**Key parameters:**

| Parameter | Default | Effect |
|-----------|---------|--------|
| `--threshold` | -40dB | Lower = more sensitive |
| `--duration` | 0.5s | Minimum silence to catch |
| `--merge` | 0.3s | Merge nearby silences |

---

## 🗣️ Skill 3: Chinese Filler Removal

---

### `remove-filler-zh` — Whisper-Powered Precision

**Word-Level Only** 🎯

Uses Whisper's `word_timestamps=True` for exact filler positions. No segment-level fallback estimation.

**Filler words** (13 + 8 phrases):

| Type | Examples |
|------|----------|
| Single-char | 呃, 啊, 嗯, **呢**, **嘞** |
| Phrases | 那个, 就是, 然后, 这个 |

**Detection logic:**
```
For each word from Whisper:
  if word.text ∈ FILLER_WORDS → remove
  if phrase ∈ word.text → remove
Merge overlapping ranges
```

]

<pre>
graph TD
 VID["🎬 Input Video"] --> AUD["🎵 Extract Audio 16kHz WAV"]
 AUD --> WH["🧠 Whisper word_timestamps=True"]
 WH --> WL["Word List (text, start, end)"]
 WL --> FD["Filler Detection word-level match"]
 FD --> RM["Filler Ranges to remove"]
 RM --> IV["Invert → Keep Ranges"]
 IV --> FF["🎬 FFmpeg filter_complex trim + concat"]
 FF --> OUT["🎬 Clean Video"]
 style VID fill:#bf616a,color:#eceff4
 style WH fill:#5e81ac,color:#eceff4
 style FD fill:#b48ead,color:#eceff4
 style FF fill:#d08770,color:#eceff4
 style OUT fill:#a3be8c,color:#2e3440
</pre>

]

---

### ⚖️ Word-Level vs Segment-Level

| Aspect | Word-Level ✅ | Segment-Level ❌ |
|--------|:------------:|:----------------:|
| **Precision** | Exact timestamps from Whisper | Proportional estimation within segment |
| **Accuracy** | High — cuts at exact word boundaries | Low — approximate position |
| **False positives** | Rare | Common (adjacent words caught) |
| **Coverage** | Depends on Whisper word reliability | Always covers all text |
| **Our choice** | ✅ **`remove-filler-zh`** | ❌ Not used |

**Why word-level is better for Chinese:** Recent Whisper versions (large-v3, turbo) have significantly improved Chinese word timestamp accuracy, making segment-level fallback unnecessary.

---

### 🔪 Precision Cutting with `filter_complex`

Uses FFmpeg's `filter_complex` with parallel `trim`/`atrim` filters for each keep segment:

```
[0:v]trim=0:12.5,setpts=PTS-STARTPTS[v0];
[0:a]atrim=0:12.5,asetpts=PTS-STARTPTS[a0];
[0:v]trim=14.2:30.1,setpts=PTS-STARTPTS[v1];
[0:a]atrim=14.2:30.1,asetpts=PTS-STARTPTS[a1];
...
[v0][a0][v1][a1]...concat=n=N:v=1:a=1[v][a]
```

**Why `filter_complex` is better** than `select`/`aselect`:
- Cuts at **exact timestamps** (not frame boundaries)
- Handles **arbitrary numbers** of keep segments efficiently
- Parallel execution of trim operations

---

## 💡 Key Challenges Solved

---

### 🪟🐛 Windows Temp File Bug

**Problem:** `tempfile.NamedTemporaryFile(delete=False)` on Windows leaves an open file handle that can interfere with FFmpeg writing to the same path.

**Symptoms:**
```
[in#0] Impossible to open '..._trimmed.mp4'
Error opening input: No such file or directory
```

**Root Cause:** On Windows, `NamedTemporaryFile` opens the file with handle-sharing semantics that prevent FFmpeg subprocesses from writing to it, even with `delete=False`.

**Fix:** Replace with `tempfile.mkstemp()`:
```python
def _mktemp(suffix):
    fd, path = tempfile.mkstemp(suffix=suffix)
    os.close(fd)  # Close handle immediately
    return path   # File persists, freely writable
```

---

### ⏱️ Timeout Estimation

**Problem:** Video encoding speed varies wildly (0.6x–1.3x realtime). Guessing timeouts causes failures.

**Solution:** Print estimated total time **before starting**, calculated from conservative speed assumptions:

```
Input: deepseek_v4_flash.mp4 (1116s)
Chunks: 3 (max 420s each)
Estimated total: ~2258s (38 min)
  (encode at 0.5x assumed speed; set timeout >= 38 min)
```

**Formula:**
```
estimated = duration / assumed_speed + chunks × overhead + safety_margin
```

**Per-skill estimates:**

| Skill | Bottleneck | Assumed Speed |
|-------|-----------|:------------:|
| `remove-silent-gaps` | FFmpeg re-encode | 0.5× realtime |
| `remove-filler-zh` | Whisper transcription | 1.5× realtime |

---

### 🎯 Pipeline Composition

**Processing order matters:**

<pre>
graph LR
 RAW["🎬 Raw 1116s"] --> SRT["📝 Fix SRT Subtitle Fixes"]
 SRT --> SIL["🔇 Silence Removal remove-silent-gaps"]
 SIL --> FIL["🗣️ Filler Removal remove-filler-zh"]
 FIL --> FINAL["🎬 Final 777s (-30%)"]
 
 style RAW fill:#bf616a,color:#eceff4
 style SRT fill:#5e81ac,color:#eceff4
 style SIL fill:#b48ead,color:#eceff4
 style FIL fill:#d08770,color:#eceff4
 style FINAL fill:#a3be8c,color:#2e3440
</pre>

]

**Why silence first, then fillers?**
1. **Silence removal** re-encodes video (libx264) — lossy
2. **Filler removal** also re-encodes — doing silence last would re-encode twice
3. Silence removal reduces video length, making Whisper transcription **faster**

---

## 📊 Results

---

### 📈 Before & After

**deepseek_v4_flash.mp4** (18.6 min lecture)

| Stage | Duration | Change |
|-------|:--------:|:------:|
| 🎬 Original | 1116s | — |
| 🔇 After silence removal | 902s | **−19%** |
| 🗣️ After filler removal | 777s | **−14%** |
| **🎯 Total** | **777s** | **−30%** |

**3 skills created:**

- `fix-srt` + `validate-srt` — SRT subtitle fixes
- `remove-silent-gaps` — split/trim/concat pipeline
- `remove-filler-zh` — word-level Chinese filler removal

]

**Visual Timeline:**

<pre>
gantt
 title Video Duration Comparison
 dateFormat X
 axisFormat %s
 
 section Duration
 Original :a1, 0, 1116
 After Silence :a2, 0, 902
 After Fillers :a3, 0, 777
 
 section Savings
 Silence Removed :b1, 902, 1116
 Fillers Removed :b2, 777, 902
</pre>

]

**Savings breakdown:**
- 🎯 **214s** of dead air removed (silence detection)
- 🎯 **125s** of filler words removed (Whisper + word-level)
- 🎯 **339s total** — the video is now **30% shorter** and much tighter!

]

---

### 🔧 Skills Architecture

<pre>
graph TD
 SK["⚙️ OpenCode Skills"] --> SRT["📝 fix-srt SRT transcription fixer"]
 SK --> VAL["✅ validate-srt SRT validator"]
 SK --> SG["🔇 remove-silent-gaps Split + dead air removal"]
 SK --> FZ["🗣️ remove-filler-zh Chinese filler removal"]
 
 SG --> DA["🔧 remove-dead-air (dependency)"]
 
 style SK fill:#5e81ac,stroke:#88c0d0,color:#eceff4
 style SRT fill:#bf616a,color:#eceff4
 style VAL fill:#a3be8c,color:#2e3440
 style SG fill:#b48ead,color:#eceff4
 style FZ fill:#d08770,color:#eceff4
 style DA fill:#4c566a,color:#eceff4
</pre>

]

**4 new skills + 1 dependency:**
- All bundled as `SKILL.md` + Python scripts in `~/.config/opencode/skills/`
- Cross-skill dependency: `remove-silent-gaps` calls `remove-dead-air` via subprocess
- Each skill has **timeout estimation** printed at start
- Windows temp files use `mkstemp` (no `NamedTemporaryFile`)

---

### ⚡ Performance Notes

**Processing time breakdown** (for 1116s video on CPU):

| Phase | Time | Bottleneck |
|-------|:----:|:----------:|
| Split (3 chunks) | ~2s | I/O bound, stream copy |
| Silence removal × 3 | ~30 min | FFmpeg re-encode (0.6× speed) |
| Concat | ~1s | I/O bound, stream copy |
| **Silence total** | **~31 min** | |
| Audio extract | ~5 min | FFmpeg PCM conversion |
| Whisper transcribe (medium) | ~16 min | CPU-bound, ~94 frames/s |
| Filler cut | ~3 min | FFmpeg re-encode |
| **Filler total** | **~24 min** | |
| **Grand total** | **~55 min** | For **18 min** of video |

---

## 🏁 Key Takeaways

---

### 💡 What We Learned

**Technical Lessons:**

1. 🪟 **Windows temp files are tricky** — `NamedTemporaryFile` handles persist beyond the object lifetime. Use `mkstemp` + immediate `close()` for subprocess output files.

2. ⏱️ **Don't guess timeouts** — Calculate and print estimates. Video encoding can vary 2× in speed depending on content complexity.

3. 📊 **Pipeline order matters** — Silence removal first reduces video length, making subsequent Whisper transcription faster and cheaper.

4. 🎯 **Word-level > segment-level** — Modern Whisper models have reliable Chinese word timestamps. Segment-level proportional estimation is unnecessary.

]

**Skills Delivered:**

| Skill | Lines | Purpose |
|-------|:-----:|---------|
| `fix-srt` | 189 | SRT transcription fixes |
| `validate-srt` | 122 | SRT structure validation |
| `remove-silent-gaps` | 129 | Split + trim dead air |
| `remove-filler-zh` | 175 | Whisper word-level filler removal |

**30% shorter videos** with cleaner audio and professional pacing.

]

---

### 🚀 Future Improvements

**Performance:**
- 🎮 GPU acceleration for Whisper (CUDA) → 5-10× faster transcription
- ⚡ `--preset ultrafast` for silence removal previews
- 🔄 Parallel chunk processing for silence removal

**Features:**
- 🌐 Multi-language filler word support
- 📊 HTML report with before/after metrics
- 🎯 Confidence-based filtering (only remove fillers above confidence threshold)

]

**Integration:**
- 🔗 Chained pipeline command (one-shot: silence + fillers + SRT)
- 📦 pip package for the tools
- 🤖 GitHub Actions CI for subtitle validation

]

---

### 📚 References

- **OpenAI Whisper**: [github.com/openai/whisper](https://github.com/openai/whisper) — Speech recognition with word-level timestamps
- **FFmpeg silencedetect**: [ffmpeg.org](https://ffmpeg.org/ffmpeg-filters.html#silencedetect) — Audio silence detection filter
- **Remark.js**: [remarkjs.com](https://remarkjs.com) — Presentation framework used for this slideshow

---

## 👋 Thank You!

### Happy Video Processing! 🎬 ✨

*Questions?* 🎤