Skip to main content
AI/ML Evaluation Framework

Scope

Category: Evaluation Testbeds

This page sets out what the AI/ML Evaluation Framework covers: its current focus areas, the repositories it is built from, and how it relates to 5G-MAG's standards work.

What This Project Implements​

note

This introduction reflects 5G-MAG's current focus areas for AI/ML and may evolve as the work progresses.

The AI/ML Evaluation Framework provides open-source reference tools for assessing how Artificial Intelligence and Machine Learning models behave in media-over-5G contexts. Rather than producing standards text, the project builds practical tooling (benchmarks, datasets, and traffic-evaluation harnesses) that lets researchers and engineers measure AI/ML performance for media use cases under realistic, reproducible conditions.

Current focus areas include:

  • Evaluating ML models for media quality enhancement and content-aware encoding
  • Adaptive bitrate (ABR) optimisation informed by network-side and UE-side data
  • Traffic classification and characterisation for AI-driven media services
  • UE data collection to support model training and inference in media applications

The hands-on traffic characterisation work (network emulation and an AI traffic testbed) currently lives in the 6G Testbed reference implementation, which the AI/ML activity shares and builds on. See AI Traffic Characterization for the testbed itself.

It helps to be precise about the boundary between this project and 3GPP.

  • This project does not implement a 3GPP specification. It is an evaluation harness that measures how AI/ML models and AI-driven media services behave over realistic network conditions, so results can inform standards work.
  • The 3GPP data-collection pipeline is implemented elsewhere. The normative Data Collection AF from SA4 (3GPP TS 26.531 and TS 26.532) is implemented in the separate Data Collection project, not here. This project can consume or complement that data, but it does not reimplement the AF.
  • The network-conditioning and AI-traffic measurement lives in the 6G Testbed. The hands-on tooling (network emulator plus AI traffic characterisation) is the 6G Testbed, which this activity shares. That is the code you run.

In short: the standards define the scenarios and interfaces; the 6G Testbed lets you reproduce network conditions and measure behaviour; and this AI/ML activity is the framing that turns those measurements into evaluation of AI/ML for media.

These tools are complementary to, not a replacement for, the standardisation activity. 5G-MAG contributes media-specific AI/ML requirements and use cases to 3GPP (primarily SA4 for media aspects, with SA2 for network data analytics), while the reference tools provide an experimental platform to validate and inform that work. In practice the relationship is two-way: the studies and specifications (for example 3GPP TR 26.927, the Release 19 study on AI/ML in 5G media services, its companion Release 19 evaluation report TR 26.847, and the SA4 data collection specifications TS 26.531 / TS 26.532) define the scenarios and interfaces the tools evaluate, and results from the tools can feed back into 5G-MAG positions and future contributions.

Key GitHub repositories​

The AI/ML reference tooling is currently developed across the following 5G-MAG repositories. The live, auto-generated list (including auxiliary repositories) is on the Resources page.

  • 5G-MAG/6G-Testbed: network emulator and AI traffic characterisation testbed used for AI/media traffic evaluation
  • 5G-MAG/Standards: tracking of 5G-MAG contributions to 3GPP, including AI/ML-related items

The data collection components that feed AI/ML training and inference (3GPP TS 26.531 and TS 26.532) are implemented in the separate Data Collection project (5G-MAG/rt-data-collection-application-function).

info

Some repositories may be private and under testing. Request early access.

Getting Started​

Because the runnable code is the shared testbed, the quickest path is to start there:

  1. Read the 6G Testbed scope to understand the two building blocks (network emulator and AI traffic characterisation framework).
  2. Follow the Tutorials, which walk through the shared 6G Testbed tooling.
  3. Clone 5G-MAG/6G-Testbed and run a scenario against an emulated profile, then inspect the logged metrics.

For the data-collection components (TS 26.531 / TS 26.532), see the Data Collection project and its repository 5G-MAG/rt-data-collection-application-function.

Roadmap​

General Project Roadmap​

The Kanban board below tracks the planned and in-progress work for the AI/ML Evaluation Framework (benchmarks, datasets, and traffic-evaluation tooling). It is maintained live on GitHub.

Kanban board: Roadmap

note

Refer to the Standards repository to contribute to this documentation.