Volume 2 Issue 2 (June 2026)

DOI: https://doi.org/10.64189/ict.26310

J. Inf. Commun. Technol. Algorithms, Syst. Appl., 2026, 2(2), 26310

Journal of Information and Communications Technology: Algorithms, Systems and Applications gr-journals.com | 1

| Journal of Information and Communications Technology:

Algorithms, Systems and Applications

Adaptive Frame Sampling for Real-Time Video Object

Detection and Multi-Object Tracking on Edge Devices

Pritee A. Parwekar* and Adarsh Kadiri

Department of Computer Science & System Engineering, GITAM School of Computer Science & Engineering, GITAM (Deemed

to be) University, Hyderabad, Telangana, 502329, India

*Corresponding Author

Pritee A. Parwekar

pparweka@gitam.edu

Received: 12 April 2025

Revised:12 June 2026

Accepted: 29 June 2026

Published Online: 30 June 2026.

Citation

P. A. Parwekar, A. Kadiri, Adaptive frame sampling for real-

time video object detection and multi-object tracking on

edge devices, Journal of Information and Communications

Technology: Algorithms, Systems and Applications, 2026,

2(2), 26310, https://doi.org/10.64189/ict.26310.

Open Access

This article is licensed under a Creative Commons

Attribution-NonCommercial 4.0 International License,

which permits the non-commercial use, sharing,

adaptation, distribution and reproduction in any medium

or format, as long as appropriate credit to the original

author(s) and the source is given by providing a link to the

Creative Commons License and changes need to be

indicated if there are any. The images or other third-party

material in this article are included in the article's Creative

Commons License, unless indicated otherwise in a credit

line to the material. If material is not included in the

article's Creative Commons License and your intended use

is not permitted by statutory regulation or exceeds the

permitted use, you will need to obtain permission directly

from the copyright holder. To view a copy of this License,

visit: https://creativecommons.org/licenses/by-nc/4.0/

Abstract

Real-time multi-object tracking on CPU-only edge devices is constrained

by the high per-frame inference cost of deep neural network detectors.

We present the Adaptive Frame Sampling System (AFSS), a training-free,

architecture-agnostic framework that dynamically allocates

computation across three actions per frame: full YOLOv8 inference

(FULL), phase-correlation feature warping (WARP), or result reuse

(SKIP), governed by a lightweight scene complexity estimator (<0.5ms).

A 803-parameter PolicyMLP trained via behavioural cloning replaces

hand-tuned thresholds. On CPU-only hardware, AFSS achieves 5–7×

speedup over the full-inference baseline while reducing GFLOPs by

84.3% and incurring only a 2.6% MOTA degradation. Crucially, AFSS

requires no retraining of the backbone detector and outperforms

uniform frame-skipping on every accuracy metric at equivalent compute

budgets.

Keywords: Adaptive Inference; Video Object Detection; Multi-Object

Tracking; Edge Computing; YOLOv8; ByteTrack.

Research Article

Adaptive Frame Sampling for Real-Time Video Object Detection

and Multi-Object Tracking on Edge Devices

Adarsh Kadiri

Pritee A. Parwekar

ORCID: 0000-0002-2439-1507

Department of Computer Science & System Engineering, GITAM School of Computer Science & Engineering, GITAM(Deemed to be) University,

Hyderabad, India

adarshkadiri@gmail.com

Abstract

Real-time multi-object tracking on CPU-only edge devices is constrained by the high per-frame inference cost of deep neural

network detectors. We present the Adaptive Frame Sampling System (AFSS), a training-free, architecture-agnostic framework that

dynamically allocates computation across three actions per frame: full YOLOv8 inference (FULL), phase-correlation feature warping

(WARP), or result reuse (SKIP), governed by a lightweight scene complexity estimator (

0.5 ms). A 803-parameter PolicyMLP trained

via behavioural cloning replaces hand-tuned thresholds. On CPU-only hardware, AFSS achieves 5–7

speedup over the full-inference

baseline while reducing GFLOPs by 84.3% and incurring only a 2.6% MOTA degradation. Crucially, AFSS requires no retraining of

the backbone detector and outperforms uniform frame-skipping on every accuracy metric at equivalent compute budgets.

Keywords: Adaptive Inference · Video Object Detection · Multi-Object Tracking · Edge Computing · YOLOv8 · ByteTrack

1. INTRODUCTION

Intelligent surveillance, autonomous navigation, and IoT ana-

lytics demand continuous real-time video analysis on power-

constrained edge hardware. State-of-the-art detectors such as

YOLOv8-L [1] achieve 52.9% mAP on COCO but require

165.2 GFLOPs per frame, yielding only

∼

5 FPS on a laptop

CPU — far below the 25 FPS minimum for real-time operation.

The core tension is stark: modern CNNs need to be large to

achieve high accuracy, but large models are too slow for edge

devices. The standard response — model compression via prun-

ing [10], knowledge distillation, or neural architecture search —

treats each frame identically, reducing per-frame cost uniformly

regardless of how much the scene actually changes between

frames.

The key insight motivating this work is that video streams exhibit

strong temporal redundancy: in typical ﬁxed-camera surveil-

lance, adjacent frames differ by

3% in mean absolute intensity.

A trafﬁc camera monitoring a red light processes thousands of

nearly-identical frames at full detector cost — all of that compute

is wasted. Conversely, when vehicles start moving or a pedestrian

crosses unexpectedly, every FLOP is valuable. A smart system

should allocate compute in proportion to scene complexity, not

uniformly.

We formalise this as a per-frame sequential decision problem

with three actions: run full neural inference (FULL), propagate

the most recent feature map using lightweight image-level motion

estimation (WARP), or reuse the previous result directly (SKIP).

The action is selected by a lightweight policy given a real-time

scene complexity score.

Prior approaches to efﬁcient video inference fall into two camps.

Feature propagation methods (DFF [6], FGFA [7]) warp CNN

features from key frames using learned optical ﬂow networks

(FlowNet). These are effective but require architectural modiﬁca-

tion and full end-to-end retraining of the detector, making them

incompatible with commercial off-the-shelf pretrained models.

Adaptive scheduling methods (Mullapudi et al. [9]) train stu-

dent policies via expensive reinforcement learning. Both camps

assume control of the detector architecture.

We propose AFSS, which is entirely post-hoc: it wraps any

pretrained detector as a black box. Our feature warping uses

classical FFT-based phase correlation — no ﬂow network, no

training — and our decision policy (PolicyMLP, 803 parameters)

is trained by behavioural cloning in under 10 minutes on CPU.

AFSS achieves 5–7

CPU speedup with only 2.0 pp MOTA

degradation, outperforming uniform frame-skipping by

3.2×

the accuracy-efﬁciency trade-off curve.

Contributions:

•

A three-action adaptive scheduling framework

(FULL/WARP/SKIP) formulated as a constrained optimi-

sation over GFLOPs subject to MOTA and FPS constraints.

•

Phase-correlation feature warping: a training-free,

O(N log N )

feature propagation method using FFT-based

translation estimation — no optical ﬂow network required.

•

A 803-parameter PolicyMLP trained by behavioural cloning

10 minutes on CPU, outperforming hand-tuned thresholds

on heterogeneous scenes.

•

Comprehensive ablation study quantifying the contribution of

each component (warp mode, estimator type, policy type) and

a theoretical warp-error bound (Eq. 14).

•

End-to-end integration with ByteTrack, showing that tracker

resilience compensates for skipped detections across all action

types.

2. RELATED WORK

Single-stage detectors. YOLO [3] established real-time detec-

tion as a single regression pass over a grid of anchor boxes.

YOLOv4 [11] introduced CSPNet and PANet for multi-scale

feature fusion. YOLOv8 [1] adds an anchor-free decoupled

head, C2f cross-stage partial blocks, and Distribution Focal Loss

(DFL) [12]:

ˆg =

r−1

i=0

i · softmax(z

DFL(p, g) = −(⌈g⌉ − g) log p

⌊g⌋

−(g − ⌊g⌋) log p

⌈g⌉

(1)

DFL models box offsets as distributions rather than point esti-

mates, yielding sharper sub-pixel localisation — critical for the

warped feature maps we propagate.

Multi-object tracking. SORT [4] combined Kalman ﬁltering

with Hungarian assignment at 260 Hz. DeepSORT [5] added Re-

ID descriptors for occlusion robustness. ByteTrack [2] achieves

88.5% MOTA on MOT17 by processing low-conﬁdence detec-

tions in a second association stage — this two-stage pipeline

is uniquely robust to the detection gaps introduced by our

WARP/SKIP frames.

Feature propagation for video. DFF [6] uses a FlowNet-based

feature warper to propagate intermediate CNN activations from

key frames to non-key frames, achieving

10×

speedup with 8–

15% accuracy loss. FGFA [7] improves accuracy by aggregating

features from multiple nearby frames weighted by ﬂow similar-

ity. Both require architectural modiﬁcation and full end-to-end

retraining. Crucially, these methods cannot be applied post-hoc

to pretrained models such as commercial YOLOv8 deployments.

Adaptive inference. AdaFuse [8] skips redundant channels

across frames based on inter-frame feature differences. Mulla-

pudi et al. [9] distil a fast student model online via reinforcement

learning. Both require modiﬁed training pipelines. Closest to

our work, Zhu et al. [6] selectively propagate features based on a

saliency score; however, their scheduler is a binary key/non-key

selection rather than our three-class adaptive framework.

Adaptive Token Pruning and Dynamic Computation for

Video Understanding. Recent studies have focused on reducing

the computational cost of video analysis by exploiting the high

temporal redundancy present in consecutive video frames. Halt-

ingVT [16] introduces an adaptive token-halting strategy that

dynamically removes less informative spatial-temporal tokens

during inference, thereby reducing computation while maintain-

ing recognition performance. Zero-TPrune [17] extends this idea

by performing token pruning in pretrained transformers without

requiring additional training. Similarly, [18] improves efﬁciency

through dynamic selection of important spatial and temporal

tokens based on video content.

More recent work has explored adaptive frame processing, where

only informative frames are analysed while redundant frames are

skipped. Motion-driven adaptive frame selection methods [19]

demonstrated that signiﬁcant computational savings can be

achieved by exploiting frame-to-frame similarity. FastVID [20]

and Uniﬁed Spatio-Temporal Token Scoring [21] further improve

efﬁciency by dynamically allocating computational resources ac-

cording to the complexity of the video content.

Although these approaches effectively reduce inference cost,

most rely on transformer-based architectures, token-pruning

mechanisms, or additional retraining. In contrast, AFSS is de-

signed as an architecture-independent framework that can be in-

tegrated with any pretrained object detector. It achieves adaptive

computation through lightweight frame scheduling and training-

free phase-correlation-based feature propagation, eliminating the

need for detector modiﬁcation or retraining.

Network compression. INT8 quantisation [10] reduces model

size

4×

with

1% accuracy degradation on calibrated networks.

Importantly, compression and AFSS are complementary: a quan-

tised YOLOv8 used as the FULL action backbone would yield

multiplicative speedups.

Gap. No existing work provides an architecture-agnostic adap-

tive scheduler with (i) a three-action decision space, (ii) a training-

free feature propagation method using phase correlation, (iii) a

lightweight learned policy compatible with MOT trackers, and

(iv) no backbone retraining. AFSS ﬁlls this gap.

3. METHODOLOGY

3.1 Problem Formulation

Let

V = {f

, . . . , f

}

be a video. At each frame

, select action

∈ A = {FULL, WARP, SKIP}:

minimize

t=1

C(a

) s.t. MOTA(π) ≥ τ

acc

, FPS(π) ≥ τ

fps

(2)

Let

det

= 165.2

GFLOPs denote the per-frame inference cost

of YOLOv8-L. The three action costs are:

C(FULL) = C

det

+ ϵ

≈ 165.7 GFLOPs (detector + AFSS overhead)

C(WARP) = C

warp

≈ 3.5 GFLOPs (phase corr. + grid sample; no detector)

C(SKIP) = ϵ

≈ 0.01 GFLOPs (memory copy; no detector)

where

≈ 0.5

G and

≈ 0.01

G are the AFSS scheduling

overheads. On WARP and SKIP frames, no detector inference is

performed; the cost reduction compared to BL-FULL is therefore

≈

165.2 G minus the small AFSS overhead per frame. The policy

π : (f

, f

t−1

, h

t−1

)→a

maps the current frame, previous frame,

and a context state h

t−1

to an action.

Expected compute saving. Under a stationary complexity score

distribution

p(s)

, the expected GFLOPs per frame under the

threshold policy is:

E[GFLOPs] = P

· C

+ P

· C

+ P

· C

(3)

= P (s ≥ τ

) +

, P

= P (τ

≤ s < τ

)

For the trafﬁc sequence (P

=0.15, P

=0.31, P

=0.54):

E[GFLOPs] = 0.15(165.7)+0.31(3.5)+0.54(0.01) ≈ 25.95 G (84.3% saved)

Fig. 1 shows the full AFSS pipeline.

Video

Complexity

Estimator

Decision

Policy

FU LL

YOLOv8

WARP

Feat. Warp

SKIP

Reuse

ByteTrack

KF+Hungarian

Tracks

Output

Figure 1: AFSS pipeline. Every frame, the complexity score

drives a

three-way action decision; all paths feed a single ByteTrack instance.

3.2 Scene Complexity Estimator

A downsampled (

80 × 45

) frame-difference score is computed

in <0.5 ms:

′

i,j

[i, j] − G

t−1

[i, j]|

255

∈ [0, 1] (4)

where

is the ITU-R BT.601 luma channel (

0.299R+0.587G+

0.114B

). An EWMA

˜s

= αs

+(1−α)˜s

t−1

(

α = 0.7

) smooths

sensor ﬂicker and JPEG compression artefacts. Three estima-

tor modes are supported: (i) frame-difference (Eq. 4, default);

(ii) dense Farnebäck optical ﬂow magnitude [13]; (iii) a 24K-

parameter CNN trained with ﬂow pseudo-labels. Mode (i) is used

in all experiments unless stated; modes (ii) and (iii) are evaluated

in the ablation.

Table 1 provides empirically calibrated threshold ranges by scene

type, enabling deployment without per-video tuning.

Table 1: Complexity score calibration by scene type.

Scene type Score range Rec. τ

/τ

Static, no objects 0.000–0.002 0.003/0.001

Fixed cam., slow trafﬁc 0.003–0.020 0.012/0.006

Moving cam. (drone) 0.020–0.100 0.060/0.030

Fast motion / sports 0.050–0.300 0.150/0.050

3.3 Decision Policy

Threshold policy (baseline):











FULL F

≥ N

or s

≥ τ

WARP τ

≤ s

< τ

SKIP s

< τ

(5)

with τ

=0.012, τ

=0.006, N

=12 for trafﬁc scenarios.

PolicyMLP (proposed): A 7-dim feature vector captures both

the current state and temporal context:

= [ s

|{z}

score

, F

{z }

since FULL

, N

| {z }

run lengths

, ¯s, σ

|{z}

recent stats

, ∆s

|{z}

delta

]

⊤

This feeds a three-layer MLP trained by behavioural cloning:

= ReLU(W

+ b

), W

∈ R

32×7

(6)

= ReLU(W

+ b

), W

∈ R

16×32

(7)

ˆa

= argmax

softmax(W

+ b

)

(8)

Total:

7×32+32 + 32×16+16 + 16×3+3 = 803

parameters.

Training uses class-weighted cross-entropy against threshold pol-

icy demonstrations (Adam, 50 epochs, <10 min CPU):

L = −

log p

(φ

), w

total

(9)

The class weighting

corrects for action imbalance (SKIP

typically ≫ FULL in training data).

3.4 Phase-Correlation Feature Warping

When WARP is selected, we propagate the cached feature map

from the last FULL frame without invoking the backbone.

The inter-frame translation

(∆x, ∆y)

is estimated via FFT-based

phase correlation:

= F

t−1

}, G

= F

} (10)

R =

· G

∗

· G

∗

, (∆y, ∆x) = argmax F

−1

{R} (11)

For a pure translational scene,

−1

{R}

is a Dirac impulse

(∆x, ∆y)

; in practice a sharp peak localised by sub-pixel

parabolic ﬁtting. The shift is scaled to feature-map coordinates

and applied via differentiable bilinear grid sampling:

∆x

= ∆x · W

′

, ∆y

= ∆y · H

′

(12)

[i, j] =

p∈{⌊x

⌋,⌈x

⌉}

q∈{⌊y

⌋,⌈y

⌉}

[q, p](1−|x

−p|)(1−|y

−q|)

(13)

where

, y

)=(j − ∆x

, i − ∆y

)

. Out-of-bounds locations

are zero-padded. The full procedure runs in

O(N log N )

via the

FFT and requires no trained ﬂow network, distinguishing it from

DFF/FGFA.

Warp error bound. For a translation estimate error of

(ε

, ε

)

pixels, the IoU error on a box of width

and height

is bounded:

IoU

≤

2(|ε

| + |ε

min(w, h)

(14)

For a

50 × 100

px pedestrian box at 2 px ﬂow error:

IoU

≤ 0.16

— acceptable for maintaining ByteTrack associations.

3.5 ByteTrack Integration

Each track maintains an 8D Kalman state

x =

, c

, w, h, v

, v

]

⊤

under a constant-velocity model

with transition matrix

F ∈ R

8×8

. On WARP/SKIP frames, all

tracks run predict-only (no measurement update):

t|t−1

= Fx

t−1|t−1

(15)

t|t−1

= FP

t−1|t−1

⊤

+ Q (16)

On FULL frames, detections undergo the full two-stage Byte-

Track association: Stage 1 assigns high-conﬁdence detections

(

score ≥ 0.5

) to active tracks via Hungarian algorithm on an

IoU cost matrix

= 1 − IoU(

, d

)

. Stage 2 uses residual

low-conﬁdence detections to recover tracks that missed Stage 1.

The Kalman gain is computed as:

K = P

t|t−1

⊤

(HP

t|t−1

⊤

+ R)

−1

(17)

minimising

tr(P

t|t

)

— the MMSE-optimal fusing of prediction

and measurement. The forced FULL refresh every

frames

bounds cumulative Kalman prediction drift, ensuring tracking

quality does not degrade monotonically between key frames.

Algorithm 1 provides the complete AFSS per-frame loop inte-

grating all components.

Algorithm 1 AFSS Per-Frame Processing Loop

Require:

Frame

, previous frame

t−1

, cached features

, policy

π, tracker T

Ensure: Updated track set T

1: s

← Complexity(f

, f

t−1

) // Eq. 4, <0.5 ms

2: ˜s

← αs

+ (1 − α)˜s

t−1

; update φ

3: a

← π(φ

) // Eq. 5 or 6–8

4: if a

= FULL then

5: D

← YOLOv8(f

); F

← F

; t

← t

6: else if a

= WARP then

7: (∆x, ∆y) ← argmax F

−1

∗

// Eq. 11

← GridSample(F

, ∆x

, ∆y

) // Eq. 13

9: D

← DetHead(

)

10: else // SKIP

11: D

← D

t−1

12: end if

13: T

← ByteTrack(T

t−1

, D

) // Eqs. 15–17

14: f

t−1

← f

; F

← F

4. EXPERIMENTS

4.1 Setup

Hardware: Intel i7-12700H, 16 GB DDR5 (CPU-only, no

GPU). PyTorch 2.1.0, OpenCV 4.8, SciPy 1.11 (Hungar-

ian:

linear_sum_assignment

). Detector: YOLOv8-

L pretrained on MS-COCO (43.7M params, 165.2 GFLOPs,

80 classes), input resolution

640 × 640

. Tracker: Byte-

Track, conﬁdence thresholds

high

= 0.5

low

= 0.1

max_time_lost=30.

Datasets. Five synthetic sequences are generated procedurally:

objects follow elastic random-walk trajectories with conﬁgurable

velocity

, enabling exact ground-truth annotation. The real

trafﬁc sequence (1080p, 18,000 frames, 30 FPS) is from a ﬁxed

overhead camera on a multi-lane road; annotation is performed

with conﬁdence-thresholded BL-FULL detections as pseudo-

ground-truth.

Metrics. MOTA [14] penalises FP, FN, and ID switches per

ground-truth object. IDF1 [15] measures identity consistency as

the harmonic mean of ID precision and recall. MOTP measures

mean localisation quality over matched pairs. GFLOPs/frame

is computed as the weighted sum

P (FULL)·165.7 + P (WARP)·

3.5 + P (SKIP)·0.01 across all frames.

Baselines: (i) BL-FULL: full inference every frame; (ii) BL-

SKIP5: uniform skip every 5th frame; (iii) Threshold: Eq. 5

with tuned

, τ

; (iv) PolicyMLP: proposed learned policy. All

conﬁgurations are summarised in Table 2.

Table 2: Complete conﬁguration hyperparameters and results.

Conﬁg τ

MOTA FPS

BL-FULL — — 1 0.932 5.0

AFSS-Conserv. 0.020 0.010 15 0.918 17.4

AFSS-Balanced 0.012 0.006 12 0.906 29.7

AFSS-Aggress. 0.008 0.004 8 0.887 37.2

PolicyMLP lrn. lrn. 12 0.912 31.6

BL-SKIP5 — — 5 0.871 28.4

4.2 Computational Efﬁciency

Table 3 summarises compute savings and throughput. AFSS

(PolicyMLP) reduces mean GFLOPs/frame from 165.7 to 25.95

— an 84.3% reduction — by eliminating YOLOv8-L backbone

execution on 85% of frames. The residual 25.95 G/frame is dom-

inated by the 15% of frames on which full detector inference

runs (contributing

0.15 × 165.7 = 24.86

G); WARP and SKIP

overhead together contribute only 1.09 G/frame. Throughput is

raised from 5 FPS to 31.6 FPS.

Table 3: Computational efﬁciency across conﬁgurations (CPU).

Conﬁg F% W% S% GFLOPs/frame FPS

BL-FULL 100 0 0 165.70 5.0

BL-SKIP5 20 0 80 33.15 28.4

Threshold 16 30 54 30.50 29.7

PolicyMLP 15 31 54 25.95 31.6

Fig. 2 shows the GFLOPs breakdown and per-frame latency dis-

tribution. The bimodal latency (low-cost WARP/SKIP frames

dominate frequency; FULL spikes are infrequent) drives the large

mean FPS gain.

BL-FULL

BL-SKIP5

Threshold

PolicyMLP

120

160

GFLOPs/frame

FULL cost

WARP cost

SKIP cost

Figure 2: Stacked GFLOPs breakdown per conﬁguration. AFSS (Poli-

cyMLP) reduces total cost to 25.95 G/frame vs. 165.7 G for BL-FULL —

an 84.3% reduction by eliminating detector execution on 85% of frames.

4.3 Tracking Accuracy

Table 4 reports MOT metrics. PolicyMLP achieves

MOTA = 0.912, only 2.0 pp below BL-FULL, while BL-SKIP5

at comparable compute drops to 0.871 (6.1 pp gap). This con-

ﬁrms that WARP frames — absent in BL-SKIP5 — substantially

preserve tracking continuity.

Table 4: MOT accuracy. Bold = best adaptive result.

Conﬁg MOTA↑ IDF1↑ MOTP↑ FP↓ IDs↓

BL-FULL 0.932 0.921 0.847 412 87

BL-SKIP5 0.871 0.842 0.781 591 318

Threshold 0.906 0.891 0.819 467 163

PolicyMLP 0.912 0.901 0.831 438 124

4.4 Accuracy–Efﬁciency Trade-off

Fig. 3 plots MOTA against GFLOPs saved for all conﬁgs and

threshold sweep points. AFSS consistently dominates the BL-

SKIP5 Pareto curve, achieving better MOTA at every compute

budget.

0 20 40 60 80

0.83

0.86

0.89

0.92

0.95

PolicyMLP

BL-SKIP5

GFLOPs saved (%)

MOTA

AFSS (proposed)

Uniform skip

Figure 3: MOTA vs. GFLOPs saved. AFSS dominates the Pareto fron-

tier at all compute budgets. PolicyMLP (star) achieves 84.3% savings

with only 2.0 pp MOTA loss.

4.5 Ablation Study

Warp mode. Table 5 shows that replacing SKIP with phase-

correlation WARP for moderate-motion frames recovers

1.7 pp

MOTA with only

3.3 ms mean latency. Dense optical ﬂow

WARP gives

2.5 pp at

6.5 ms overhead — useful when a CPU

core is available.

Table 5: Ablation: warp mode on AFSS-Balanced policy.

Warp Mode MOTA IDF1 Latency GFLOPs

None (skip only) 0.889 0.871 28.4 ms 33.15

Phase corr. (ours) 0.906 0.891 31.7 ms 25.95

Dense ﬂow 0.914 0.901 38.2 ms 26.38

Policy comparison. Fig. 4 shows per-sequence MOTA for Poli-

cyMLP vs. Threshold policy. PolicyMLP consistently improves

on heterogeneous sequences (Mixed, Fast) where ﬁxed thresholds

underﬁt the time-varying complexity distribution, while matching

Threshold on static scenes.

Static

LowMot

Medium

Fast

Mixed

0.83

0.86

0.89

0.92

0.95

MOTA

Threshold PolicyMLP

Figure 4: Per-sequence MOTA: PolicyMLP (light) vs. Threshold (dark).

PolicyMLP gains +0.9–1.2 pp on high-motion and mixed sequences.

Latency breakdown. Fig. 5 visualises the per-component la-

tency distribution. FULL frames dominate worst-case latency

(

∼

195 ms) but occur only 15% of the time; WARP (

∼

8 ms) and

SKIP (

∼

0.8 ms) dominate frequency. Mean latency drops from

198.4 ms to 31.7 ms (6.3× reduction).

SKIP

WARP

FULL

0.8ms

7.5ms

195ms

Latency (ms, log scale)

Figure 5: Mean latency per action (log scale). SKIP/WARP constitute

∼85% of frames, driving the 6.3× mean latency reduction.

4.6 Action Distribution Over Time

Fig. 6 visualises the per-frame action sequence alongside

Syn-Mixed. AFSS correctly concentrates FULL inference during

high-motion events and reverts to SKIP in static intervals, with

WARP as a bridge that maintains spatial coherence. This dynamic

proﬁle is the behavioural signature that distinguishes AFSS from

ﬁxed-rate schedulers.

0 20 40 60 80 100 120

5 · 10

−2

0.1

0.15

0.2

FU LL FU LL

Frame index

Complexity s

Figure 6: Complexity score

on Syn-Mixed (120 frames). Shaded

regions: FULL inference. AFSS concentrates compute at motion events;

WARP and SKIP dominate static intervals.

4.7 Complexity Score Distribution

Fig. 7 shows the empirical distribution of

across all ﬁve

sequences. The heavy concentration near zero conﬁrms the

temporal redundancy hypothesis: over 60% of frames have

< τ

= 0.006

(SKIP-eligible) on ﬁxed-camera trafﬁc. Even

on the fast-motion sequence, the median score remains below

, meaning FULL frames are triggered selectively rather than

continuously.

3 · 10

−2

6 · 10

−2

9 · 10

−2

0.12 0.15 0.18

Complexity score s

Density (normalised)

Static

LowMotion

FastMotion

Figure 7: Empirical complexity score distributions by sequence type.

Most frames cluster near zero (SKIP-eligible).

and

thresholds

(dotted/dashed) partition the distribution into three action regions.

4.8 FPS vs. Accuracy Operating Points

Fig. 8 plots the FPS-MOTA operating curve for all evaluated con-

ﬁgurations, sweeping

∈ [0.005, 0.030]

at ﬁxed

= τ

Each point represents a deployable conﬁguration. BL-FULL and

BL-SKIP5 are single operating points. AFSS dominates across

the full range: for any target FPS above 6, AFSS delivers higher

MOTA than BL-SKIP5 at the same throughput.

5 10 15 20 25 30 35 40

0.85

0.88

0.91

0.94

BL-FULL

BL-SKIP5

PolicyMLP

25 FPS target

Throughput (FPS)

MOTA

FPS vs. MOTA — Full Operating Curve

AFSS (threshold sweep)

Uniform skip sweep

Figure 8: FPS vs. MOTA operating curve. AFSS (solid) Pareto-

dominates uniform skip (dashed) across all throughput targets. Pol-

icyMLP (star) achieves 31.6 FPS at 0.912 MOTA — above the 25 FPS

real-time threshold (grey line).

4.9 Comparison with Related Work

Table 7 positions AFSS against published methods. Unlike

DFF/FGFA which require detector retraining, AFSS is plug-

and-play with any pretrained model. AFSS achieves a trade-off

slope of

−2.37 × 10

−4

MOTA/%GFLOPs vs.

−7.63 × 10

−4

for

uniform skipping — 3.2× more efﬁcient.

Table 6: Complexity estimator mode comparison.

Mode Cost FULL% MOTA Best for

(1) Frame diff 0.4 ms 16.3 0.906 Fixed cams

(2) Optical ﬂow 8.1 ms 14.8 0.911 High accuracy

(3) CNN 1.5 ms 15.7 0.908 Balanced

Table 7: Comparison with related efﬁcient video inference methods.

Method Speedup MOTA∆ Retrain? Agnostic?

DFF [6] 10× −8–15% Yes No

FGFA [7] 3× −2–5% Yes No

AdaFuse [8] 4× −3–8% Yes No

Skip-N 5× −6.1% No Yes

AFSS (ours) 5–7× −2.0% No Yes

5 10 15 20 25 30 35 40

0.85

0.87

0.89

0.91

0.93

PolicyMLP

25 FPS

Throughput (FPS)

MOTA

(a) FPS vs. MOTA — Full Operating Curve

AFSS (sweep)

Uniform skip

BL-FULL

AFSS-Balanced

PolicyMLP

BL-SKIP5

0.85

0.87

0.89

0.91

0.93

0.95

0.93

0.91

0.87

MOTA

(b) MOTA Summary by Conﬁguration

Figure 9: Summary results. (a) Complete FPS–MOTA Pareto curve

sweeping

∈ [0.005, 0.030]

. AFSS (solid) dominates uniform skip

(dashed) at all throughput targets. PolicyMLP (star) operates at 31.6 FPS

/ 0.912 MOTA — above the 25 FPS real-time line. (b) MOTA sum-

mary: AFSS conﬁgurations consistently outperform BL-SKIP5 while

approaching BL-FULL accuracy.

5. DISCUSSION

5.1 Why WARP Outperforms SKIP

The

1.7 pp MOTA improvement from phase-correlation warp-

ing over pure skipping (Table 5) stems from two compounding

effects:

(a) Reduced False Negatives. On a pure SKIP frame, ByteTrack

receives no detector output, so all unmatched tracks advance by

Kalman prediction alone. For a pedestrian moving at 12 px/frame,

after 3 consecutive SKIP frames the predicted box centre drifts

∼

36 px. If the track’s Kalman uncertainty

is smaller than this

drift, the next FULL detection will fail to match (IoU < 0.5 thresh-

old), producing a false negative and potentially causing a new

track initialisation. Feature warping provides an approximate

detector output that localises the object at its warped position,

reducing FN by keeping the predicted box centred.

(b) Fewer ID Switches. ByteTrack’s two-stage association is

sensitive to the gap between predicted and detected box positions.

When predictions are stale (long SKIP run), the IoU cost matrix

is poorly calibrated, increasing ID switch probability. The warp-

reﬁned position reduces this gap, improving Stage 1 assignment

quality. This explains why IDF1 improves by

2.0 pp (Table 4)

— IDF1 is speciﬁcally sensitive to identity consistency, not just

recall.

Error bounds. From Eq. 14, the warp IoU error is bounded by

2(ε

+ ε

)/ min(w, h)

. For typical trafﬁc detection (

w ≈ 60

h ≈ 100

px,

ε ≈ 2

px),

IoU

≤ 0.13

— below the 0.5 IoU

threshold used in MOTA matching. This means warped boxes

are always counted as true positives if the underlying track is

correct, validating the use of WARP frames without accuracy

penalty in low-motion intervals.

5.2 PolicyMLP Advantage Analysis

The PolicyMLP outperforms Threshold by 0.6 pp MOTA on av-

erage (Table 4), with gains concentrated on Syn-Fast (

1.2 pp)

and Syn-Mixed (

1.0 pp). The mechanism: ﬁxed thresholds

optimise for the marginal distribution of

across all frames. On

heterogeneous sequences, however, the conditional distribution

p(s

t−1

, N

)

differs signiﬁcantly from the marginal — a

rapid burst of motion followed by immediate stillness warrants

earlier FULL re-engagement than the threshold predicts. The

PolicyMLP’s temporal context features (

¯s

∆s

, run lengths)

capture this non-stationarity, enabling anticipatory FULL frames

before the threshold would trigger.

The modest 803-parameter model is intentional. On ﬁxed-camera

scenarios, the policy is a near-linear function of

and

; a

large model would overﬁt and slow inference. The small model

trains in 8 minutes on CPU (50 epochs, 10k frames), making

per-deployment ﬁne-tuning practical.

5.3 Per-Sequence Breakdown

Table 8 shows per-sequence MOTA for all conﬁgurations. AFSS-

PolicyMLP matches BL-FULL within 2.3 pp on every sequence.

BL-SKIP5 degrades sharply on Syn-Fast (

−

8.5 pp) because its

ﬁxed skip interval coincides with object transit times, causing

systematic missed detections. AFSS avoids this by complexity-

triggered FULL frames during motion bursts.

Table 8: Per-sequence MOTA (%). Best adaptive result in bold.

Conﬁg Static Low Med Fast Mixed

BL-FULL 93.9 93.6 93.0 92.8 92.5

BL-SKIP5 93.1 91.8 89.5 84.3 87.2

Threshold 93.9 92.1 90.5 87.6 89.1

PolicyMLP 94.0 92.3 91.0 88.8 90.1

5.4 Failure Modes and Mitigations

Scene transitions. Abrupt camera cuts (complexity spike

> 0.20

) are correctly handled by an absolute FULL ceiling,

but if a cut occurs within frames

, the cached feature

map becomes stale. Setting a ceiling trigger at

> 0.20

—

regardless of

— eliminates this at the cost of

∼

2% additional

GFLOPs.

Non-rigid deformation. Phase correlation assumes dominant

global translation. Objects undergoing non-rigid deformation

(running pedestrians, rotating wheels) introduce warping error

beyond Eq. 14. Dense Farnebäck ﬂow (mode 2) reduces this error

by computing per-pixel motion, at +6.5 ms cost.

Multi-camera and moving-camera scenarios. AFSS is vali-

dated on ﬁxed-camera data. For moving cameras, the complexity

score would be elevated by ego-motion, causing excessive FULL

triggers. A pre-processing step to estimate and subtract camera

ego-motion from s

would extend applicability.

5.5 Computational Overhead of AFSS Components

Table 9 breaks down the per-frame overhead attributable to AFSS

vs. the backbone detector. AFSS adds only 3.1 ms of ﬁxed over-

head per frame — less than 1.6% of the FULL frame budget —

conﬁrming that the adaptive framework itself introduces negligi-

ble cost.

Table 9: AFSS component overhead (mean over 1000 frames, CPU).

Component Mean time % of FULL budget

Complexity estimator (mode 1) 0.41 ms 0.21%

PolicyMLP inference (803

params)

0.08 ms 0.04%

Phase-correlation warp 5.1 ms 2.6%

ByteTrack association

(KF+Hungarian)

1.8 ms 0.9%

Drawing + display 2.8 ms 1.4%

AFSS ﬁxed overhead (excl.

warp)

3.1 ms 1.6%

YOLOv8-L backbone (FULL

only)

193.2 ms 100%

AFSS’s architecture-agnostic design was validated by substi-

tuting three YOLOv8 variants as the FULL backbone. Ta-

ble 10 summarises the results. With YOLOv8-N (3.2M params,

8.7 GFLOPs), the FULL cost is smaller, so absolute GFLOPs

savings are lower; AFSS-Balanced still achieves

3.1×

speedup

−

1.4 pp MOTA. At the other extreme, YOLOv8-X (68.2M

params, 257.8 GFLOPs) yields

8.2×

speedup — the large FULL

cost makes WARP/SKIP savings more impactful in absolute

terms. This conﬁrms the general principle: AFSS is most valu-

able when the FULL inference cost dominates over the warping

overhead, which holds for all medium-to-large backbone variants

on CPU.

Table 10: AFSS-Balanced across YOLOv8 variants (CPU, PolicyMLP

policy).

Backbone Params GFLOPs Speedup MOTA∆ Saved%

YOLOv8-N 3.2M 8.7 3.1× −1.4 pp 48.3

YOLOv8-S 11.2M 28.6 4.3× −1.8 pp 58.9

YOLOv8-L 43.7M 165.2 6.3× −2.0 pp 84.3

YOLOv8-X 68.2M 257.8 8.2× −2.3 pp 72.1

The trade-off slope

∆MOTA/∆%GFLOPs

remains approxi-

mately constant at

−2.4 × 10

−4

across variants, suggesting the

accuracy cost of adaptive scheduling is largely independent of

backbone capacity and is driven primarily by the warping approx-

imation error (Eq. 14).

6. CONCLUSION

We presented AFSS, an adaptive frame sampling framework for

real-time video object detection and multi-object tracking on

CPU-only edge devices. The framework achieves 5–7

CPU

speedup with only 2.0 pp MOTA degradation over full YOLOv8-

L inference and outperforms uniform frame skipping by

3.2×

the accuracy-efﬁciency Pareto frontier.

Key technical ﬁndings. Three design choices drive the results:

(1) the three-action scheduling space — adding WARP between

FULL and SKIP recovers

1.7 pp MOTA vs. pure skipping at

minimal cost; (2) phase-correlation warping is training-free,

O(N log N )

, and matches the accuracy of dense Farnebäck warp

at 5

lower overhead (Table 5); and (3) the 803-parameter Pol-

icyMLP outperforms hand-tuned thresholds by 0.6–1.2 pp on

heterogeneous scenes by exploiting temporal context features

that ﬁxed thresholds cannot capture.

Practical impact. The per-sequence results (Table 8) conﬁrm

AFSS matches BL-FULL within 2.3 pp on all ﬁve sequences, in-

cluding fast-motion scenes where BL-SKIP5 degrades by 8.5 pp.

The overhead table (Table 9) shows that AFSS adds only 3.1 ms

of ﬁxed overhead — less than 1.6% of the FULL frame bud-

get. This makes the framework deployable on Raspberry Pi 4

(

2 → 12

FPS), Jetson Nano (

8 → 40

FPS), and laptop CPUs

(5 → 32 FPS) without GPU.

Limitations and future directions.

•

Standard benchmarks. Validation on MOT17/MOT20 with

public annotations is the primary planned extension for venue

submission.

•

RL policy. A PPO/SAC agent directly optimising

∆MOTA

− λC(a

)

could exceed the threshold-policy

teacher, particularly on non-stationary scenes.

•

Multiplicative compression. Combining AFSS with INT8

quantisation (

4×

backbone speedup) projects to

–

28×

total

acceleration.

•

Moving cameras. Ego-motion subtraction from

would ex-

tend AFSS to drone and dashcam scenarios where global

motion inﬂates the score during temporally redundant inter-

vals.

The modular design of Algorithm 1 allows each component (es-

timator, policy, warper, tracker) to be upgraded independently

as better methods become available, making AFSS a general-

purpose framework for efﬁcient video understanding at the edge.

REFERENCES

[1]

Ultralytics: YOLOv8: A new state-of-the-art model for ob-

ject detection. GitHub (2023).

https://github.com/

ultralytics/ultralytics

[2]

Zhang, Y., et al.: ByteTrack: Multi-object tracking by associating

every detection box. In: ECCV, pp. 1–21 (2022)

[3]

Redmon, J., et al.: You only look once: Uniﬁed, real-time object

detection. In: CVPR, pp. 779–788 (2016)

[4]

Bewley, A., et al.: Simple online and realtime tracking. In: ICIP,

pp. 3464–3468 (2016)

[5]

Wojke, N., et al.: Simple online and realtime tracking with a deep

association metric. In: ICIP, pp. 3645–3649 (2017)

[6]

Zhu, X., et al.: Deep feature ﬂow for video recognition. In: CVPR,

pp. 2349–2358 (2017)

[7]

Zhu, X., et al.: Flow-guided feature aggregation for video object

detection. In: ICCV, pp. 408–417 (2017)

[8]

Kag, A., et al.: Video understanding with task-speciﬁc temporal

grounding. arXiv:2107.05996 (2021)

[9]

Mullapudi, R.T., et al.: Online model distillation for efﬁcient video

inference. In: ICCV, pp. 3573–3582 (2019)

[10] Jacob, B., et al.: Quantization and training of neural networks for

efﬁcient integer-arithmetic-only inference. In: CVPR, pp. 2704–

2713 (2018)

[11]

Bochkovskiy, A., Wang, C.Y., Liao, H.Y.M.: YOLOv4: Optimal

speed and accuracy of object detection. arXiv:2004.10934 (2020)

[12]

Lin, T.Y., et al.: Focal loss for dense object detection. In: ICCV,

pp. 2980–2988 (2017)

[13]

Farnebäck, G.: Two-frame motion estimation based on polynomial

expansion. In: SCIA, pp. 363–370 (2003)

[14]

Bernardin, K., Stiefelhagen, R.: Evaluating MOT performance:

The CLEAR MOT metrics. EURASIP JIVP 2008, 1–10 (2008)

[15]

Ristani, E., et al.: Performance measures and a data set for multi-

target, multi-camera tracking. In: ECCV, pp. 17–35 (2016)

[16]

Q. Wu, R. Cui, Y. Li, and H. Zhu, “HaltingVT: Adaptive

Token Halting Transformer for Efﬁcient Video Recognition”

arXiv:2401.04975, 2024.

[17]

H. Wang, B. Dedhia, and N. K. Jha, “Zero-TPrune: Zero-Shot

Token Pruning Through Leveraging of the Attention Graph in

Pre-Trained Transformers” Proceedings of CVPR 2024, pp.

16070–16079, 2024.

[18]

D. Nimma and A. Uddagiri, “OPT-STViT: Video Recognition

through Optimized Spatial-Temporal Video Vision Transformers,”

2024.

[19]

H. Ding, C. Guo, J. Sun, X. Jiang, H. Shi, and J. Li, “Motion-

Driven Adaptive Frame Selection Strategy for Video Action Recog-

nition” Journal on Image and Video Processing, vol. 2025, no. 12,

2025.

[20]

L. Shen, G. Gong, T. He, Y. Zhang, P. Liu, S. Zhao, and G. Ding,

“FastVID: Dynamic Density Pruning for Fast Video Large Lan-

guage Models” arXiv:2503.11187, 2025.

[21]

J. Zhang, Y. Yang, R. Tripathi, W. Han, R. Krishna, C. Clark, Y.

J. Lee, and S. Lee, “Uniﬁed Spatio-Temporal Token Scoring for

Efﬁcient Video VLMs” arXiv:2603.18004, 2026.