Skip to main content

Short-Term Memory Reduces Semantic Redundancy in Vision-Language Activity Recognition

K. Ghamati, F. Amirabdollahian, F. Foerster & A. Zaraki • University of Hertfordshire

Under Review

About the Paper

Khashayar Ghamati, Farshid Amirabdollahian, Frank Foerster & Abolfazl Zaraki
University of Hertfordshire, UK

When vision-language models (VLMs) process video frames independently, they produce semantically redundant descriptions that fail to capture the dominant, contextually coherent activity across time. Imagine a person working at a laptop who briefly reaches for a cup of tea: a memoryless VLM classifies the interlude as drinking, missing that the sustained activity is working at a laptop. We introduce ARTEMIS (Adaptive Reasoning Through Experience Memory and Intelligent Streaming), a streaming pipeline that augments VLM-based activity recognition with an explicit short-term memory (STM) buffer, enabling contextually coherent semantic aggregation.

ARTEMIS pairs LLaVA-Next (8B) for per-frame captioning with a sliding buffer of the $W$ most recent descriptions and a Qwen-2.5-7B aggregator that consolidates the temporal context into a single activity representation. We evaluate redundancy reduction through paired within-video comparisons on two datasets: Toyota Smarthome (29 activity classes) and the Robot House Multi-View (RHM) dataset (14 classes, 28 videos). Across Toyota, STM delivers a 72.5% reduction in keyword redundancy with a 93% win rate (27/29 activities); on RHM continuous activities, STM yields an 82.1% reduction (Hedges' g = 1.29, padj = 0.024).

Key Insight

Memory-augmented VLMs retain temporal context to recognise the dominant sustained activity, suppressing brief transient interruptions. The STM buffer acts as an ICL-style contextual aggregator: the VLM and aggregator weights remain frozen, and adaptation arises entirely from the content of the sliding window.

The ARTEMIS Framework

A Streaming Pipeline with Short-Term Memory

ARTEMIS integrates three components operating in a continuous streaming loop: a vision-language perception module for per-frame captioning, a sliding short-term memory buffer for temporal context, and a semantic aggregation module that consolidates the buffer into a coherent activity representation.

ARTEMIS architecture: LLaVA-Next generates per-frame captions that accumulate in a sliding STM buffer; Qwen-2.5 aggregates the buffer into activity representations. The diagram also shows the broader cognitive framework (RLBAM attention, RL agent with transfer learning and pre-training, knowledge base) that is validated in companion papers but out of scope for this work.
Figure 1. ARTEMIS architecture. Visual observations are processed by LLaVA-Next to produce frame-level descriptions. These accumulate in the STM buffer (sliding window of $W$ frames). Qwen-2.5 aggregates the temporal context into a coherent activity representation with extracted keywords. Out-of-scope outer-loop components -- the RL-based attention mechanism (RLBAM) and the RL agent with transfer learning, pre-training and knowledge base -- are validated in our companion publications.

Component Overview

Per-Frame Perception
LLaVA-Next (llama3-llava-next-8b-hf) generates a concise textual description of each frame, focused on human posture, objects, spatial relationships, and environmental context. Descriptions are capped at 200 characters to bound memory.
Sliding STM Buffer
The $W$ most recent captions (default $W=10$, 1 s at 10 FPS) are retained in an ordered buffer. The buffer has constant memory footprint; older descriptions are displaced as new observations arrive.
Semantic Aggregation
Qwen-2.5-7B-Instruct consolidates the concatenated buffer into a coherent activity representation, identifying the primary human activity, detecting transitions, and extracting contextual keywords that summarise the sustained behaviour.
Paired Evaluation
We compare STM against a 1-frame baseline on the same video at the keyword level. Activity categories on RHM (Continuous / Fine Motor / Transitional) reveal that STM benefits are activity-dependent, with strong gains on sustained behaviours.

Step-by-Step Processing

The figure below walks through the Sense--Think--Act cycle on a real example from the RHM dataset. The SENSE stage shows LLaVA processing successive frames with a context-aware prompt that incorporates the prior STM content. The THINK stage demonstrates how frame descriptions accumulate in the buffer across ten frames, followed by Qwen-based semantic aggregation that extracts consolidated activity keywords.

Step-by-step ARTEMIS pipeline walkthrough on an RHM video: ten frames passed to LLaVA, descriptions accumulating in the STM buffer, Qwen-2.5 consolidating the buffer into activity keywords.
Figure 2. Detailed ARTEMIS processing pipeline. The Sense--Think--Act cycle is illustrated on a real RHM example. Frame-level descriptions accumulate across the 10-frame STM window, then Qwen-2.5 performs semantic aggregation to produce a contextually coherent keyword set.

Contribution

ARTEMIS is an architecturally simple, model-agnostic wrapper: the VLM and aggregator remain frozen, while temporal consolidation emerges from the buffer content. This makes the approach compatible with existing single-frame VLMs while maintaining the streaming, low-latency properties required for human-robot interaction.

Key Results

Paired Within-Video Evaluation Across Two Datasets

  • Toyota Smarthome (29 activities): STM reduces mean keyword redundancy from 0.661 to 0.182 -- a 72.5% reduction. Lower redundancy is achieved in 27 of 29 activities (93.1% win rate), and 20 of 29 activities reach complete redundancy elimination.
  • RHM continuous activities (n = 8): Walking, Drinking, Cleaning and Carrying Object show a 100% win rate and 82.1% relative reduction (Hedges' g = 1.29, padj = 0.024 after Bonferroni correction).
  • RHM transitional activities (n = 14): Sitting Down, Standing Up, Stairs Climbing, Lifting and Bending -- STM does not help and slightly increases redundancy, a pattern consistent with event segmentation theory: when the buffer spans a boundary between incompatible states, the aggregator cannot identify a single dominant activity.
  • Temporal coverage ablation: At 5 FPS (where the 10-frame buffer spans 2 seconds), STM achieves a 91.9% reduction compared to 72.5% at 10 FPS. The NoSTM baseline is stable across frame rates, indicating that the effect operates through what is accumulated in the memory buffer rather than through per-frame VLM output alone.
72.5%
Toyota redundancy reduction
93.1%
Win rate (27/29 activities)
1.29
Hedges' g, RHM continuous
91.9%
Reduction at 5 FPS

Cross-Dataset Overview

At the aggregate level, STM lowers keyword redundancy on both datasets, but the magnitude of the effect differs. Toyota (predominantly sustained activities) shows a large, consistent reduction with a 93.1% win rate; RHM (half of which is transitional) shows a smaller aggregate effect because STM helps on continuous activities and hinders on transitions.

Panel A: mean redundancy for NoSTM vs STM on Toyota Smarthome and RHM with error bars. Panel B: STM win rate for each dataset, with a dashed line at 50% chance.
Figure 3. Cross-dataset overview of STM effects. (A) Mean redundancy for the NoSTM baseline and STM-enabled conditions on Toyota Smarthome and RHM. Error bars are standard deviation. (B) STM win rate per dataset; the dashed line at 50% indicates chance performance.

Per-Activity Breakdown

Most Toyota activities achieve complete redundancy elimination under STM; 20 of 29 activities reach zero redundancy, covering all Drinking variants (except From Cup), all Eating activities, all Beverage Preparation activities, and several Cooking and Transitional activities. Two activities (Cook.Cleandishes, Drink.Fromcup) show a degenerate aggregator failure mode where the model produces repetitive templates about the absence of change.

Horizontal bar chart showing per-activity difference in redundancy (NoSTM minus STM) across both datasets. Positive values indicate STM improvement.
Figure 4. Per-activity redundancy reduction. Horizontal bar chart showing the difference in redundancy (NoSTM $-$ STM) for each activity across both datasets. Positive values (rightward) indicate an STM improvement; negative values (leftward) indicate a baseline advantage.

Activity-Dependent Efficacy on RHM

The RHM subgroup analysis reveals a clear dissociation. STM lowers redundancy for Continuous activities (100% win rate, Hedges' g = 1.29) and Fine Motor activities (83% win rate, g = 0.63), but increases redundancy for Transitional activities where the buffer spans a state change and no single dominant activity exists. This is consistent with event segmentation theory and suggests a hybrid pipeline that pairs the STM buffer with a structural-shift detector as a principled next step.

Bar chart comparing NoSTM and STM mean redundancy across three post-hoc categories on RHM: Continuous, Fine Motor, Transitional. STM lowers redundancy for Continuous and Fine Motor but raises it for Transitional.
Figure 5. Redundancy by RHM activity category. NoSTM (baseline) and STM mean redundancy across the three post-hoc categories on RHM. STM reduces redundancy for Continuous and Fine Motor activities and increases it for Transitional activities.

Bootstrap Confidence Intervals

10,000-iteration bootstrap CIs corroborate the subgroup pattern. Continuous (CI [22.8%, 54.3%]) and Fine Motor (CI [3.4%, 60.0%]) both exclude zero; Transitional (CI [-42.0%, 12.4%]) and the Overall estimate both include zero with a negative central tendency for transitional activities. Bootstrap CIs are not Bonferroni-adjusted; correction applies only to the Wilcoxon $p$-values.

Forest plot showing mean redundancy reduction and 95% bootstrap confidence intervals for each RHM activity category and overall. Continuous and Fine Motor intervals exclude zero; Transitional spans zero with a negative central tendency.
Figure 6. Bootstrap forest plot. Mean redundancy reduction with 95% bootstrap CIs (10,000 iterations) for each RHM activity category and overall. Diamond markers indicate categories whose CI excludes zero; the dashed vertical line denotes no effect.

Temporal Coverage Ablation

Frame Rate and Memory-Window Span

Lowering the sampling rate from 10 FPS to 5 FPS while holding the buffer at $W=10$ doubles the temporal span covered by the STM buffer (1 s $\to$ 2 s). At 5 FPS, STM reaches a 91.9% redundancy reduction, exceeding the 72.5% obtained at 10 FPS. Crucially, the NoSTM baseline is nearly identical at both rates (0.661 vs 0.645), providing a partial natural control: if the improvement were purely a consequence of the VLM receiving more frames, the NoSTM side would shift too. The observed pattern is consistent with the effect operating through the contents of the memory buffer rather than through per-frame VLM output alone.

Per-activity redundancy reduction comparing 5 FPS and 10 FPS on Toyota Smarthome. Most activities reach full redundancy elimination at 5 FPS.
Figure 7. Temporal coverage ablation. Per-activity redundancy reduction at 5 FPS versus 10 FPS on Toyota Smarthome. Reducing the frame rate while holding $W=10$ constant doubles the temporal extent spanned by the STM buffer, so this reflects a joint change in sampling density and temporal coverage.

Design Implication

For deployment, lower sampling rates -- together with the corresponding wider temporal window -- amplify STM's consolidation benefit and simultaneously reduce GPU load. A disentangled follow-up experiment ($W=10$ at 5 FPS vs.\ $W=20$ at 10 FPS, both spanning 2 s) is needed to separate sampling density from temporal coverage.

Scope and Caveats

We explicitly scope the empirical contribution to representation efficiency -- measured as keyword redundancy reduction -- and not to downstream activity-classification accuracy. Efficiency and accuracy are separable properties that can diverge; validating that the observed efficiency gains translate into improved classification or novelty-detection performance requires a downstream-task evaluation that we flag as the primary follow-up.

The current baseline processes a single frame while STM processes $W=10$ frames, so the observed differences confound temporal consolidation with information quantity. A matched multi-frame baseline (all $W$ frames processed independently, then aggregated) is the definitive causal test and is identified as the highest-priority next experiment.

Honest Framing

This is an exploratory, hypothesis-generating study. We report descriptive statistics on Toyota Smarthome (breadth-first sampling: one video per class across all 29 activity classes) and inferential statistics on the paired RHM design. Sample sizes on RHM subgroups (n = 8 continuous, n = 6 fine motor) are modest; confirmatory evaluation on larger samples is warranted.

Why It Matters

Streaming activity recognition is a foundational capability for autonomous systems deployed in human-centred environments -- healthcare, assisted living, and collaborative robotics. Yet modern VLMs typically process each frame independently, producing repetitive, temporally incoherent outputs that overshadow sustained activities with brief transient events. ARTEMIS advances the field by:

  • Introducing an architecturally simple, model-agnostic memory wrapper that retrofits temporal coherence onto any frame-level VLM without fine-tuning.
  • Providing empirical evidence that STM benefits are activity-dependent, with sustained (continuous) activities benefiting strongly and transitional activities requiring complementary boundary-detection mechanisms.
  • Identifying a natural bridge to the inferred-structure vs in-context learning framework: STM acts as an ICL-style contextual adaptation that dominates under contextual shifts but must be paired with structural mechanisms when tasks change mid-buffer.
  • Characterising a redundancy-efficiency-accuracy trade-off that reframes how streaming VLM outputs should be evaluated -- efficiency is measurable now; accuracy requires follow-up downstream-task evaluation.

Impact

ARTEMIS enables memory-augmented VLMs to maintain coherent activity representations across brief interruptions -- a prerequisite for activity-aware dialogue, novelty detection, and safe human-robot interaction in streaming settings.

Open Science & Collaboration

We are committed to open, reproducible research. If you are interested in collaborating, extending the framework, or accessing the experimental data and evaluation scripts, please reach out.

Full Paper & Code

Download Full Paper Access Code & Materials

Under review at Expert Systems with Applications (Elsevier). A preprint, code, and experimental materials will be made available at ghamati.com/stm-vlm/ upon acceptance.