A deep convolutional network pipeline for classifying videos as real or fake, using face-region extraction and a fine-tuned Inception-ResNet-V2 classifier.
Ayan Mitra, 2020 — audited and re-documented 2026.
The pipeline extracts frames from an input video, detects and crops face regions (OpenCV + dlib), and classifies each frame as real or fake using a CNN pretrained on ImageNet and fine-tuned on labelled deepfake video frames.
Deep convolutional network based on Inception-ResNet-V2
(from the GoogLeNet family), pretrained on ImageNet (~1M images,
164 layers, 56M parameters) and fine-tuned end-to-end (trainable=True) on frame-level real/fake
labels.
- Frame extraction: OpenCV2
- Face detection/cropping: dlib
- Pooling: Global Average Pooling · Activation: Softmax · Optimizer: Adam · Batch size: 80
200 labelled .mp4 videos per class (real/fake), a few seconds to ~2 minutes long, mostly <20MB
(up to 110MB). Frames were extracted per video, face-cropped, and pooled into flat real/ and
fake/ image folders before classification.
| Split | Accuracy |
|---|---|
| Train | 99.7% |
| Held-out | 98.5% (TP 51.87% · TN 46.63% · FP 0.89% · FN 0.60%) |
The held-out split in Sentinel.ipynb is a random train_test_split over individually extracted
frames, not over videos. Frames are pooled into flat real//fake/ folders before splitting, so
frames from the same source video can appear in both the training and held-out sets. Since
neighboring frames from one video are highly correlated (same face, lighting, compression
artifacts), this almost certainly inflates the reported 98.5% figure relative to true
video-level, unseen-source generalization — this is a more likely explanation for the accuracy/loss
gap noted below than hyperparameter choice alone.
This has not yet been corrected in this repository. The honest way to evaluate this model is a
video-level (grouped) split, e.g. GroupShuffleSplit keyed on video filename, followed by reporting
video-level metrics (majority vote or mean score per video) alongside frame-level ones. Until that's
done, treat the 98.5% figure as an upper bound, not a generalization estimate.
The original training run showed a tendency for the validation loss to diverge from training loss
(see loss2.png), suggesting some memorization even before accounting for the leakage above.
Candidate fixes: dropout, learning-rate decay on the Adam optimizer, a larger and more diverse video
sample, and a systematic hyperparameter sweep.
- Re-split at the video level and re-report both frame- and video-level metrics
- Add precision/recall/F1/ROC-AUC (accuracy alone is a weak metric for a security-relevant classifier)
- Evaluate generalization to manipulation methods/generators not seen in training
- Compare against a more modern backbone (e.g. EfficientNet, a ViT-based detector)
Sentinel.ipynb— training pipelineSentinel_II_Video_Check.ipynb— run the trained model against a new video
Example: a tested video and its real/fake verdict.
Celeb-DF / benchmark comparison of deepfake-detection architectures — useful context for where this approach sits relative to current methods.