See also: Developer Exchange.
Category: Evaluation Testbeds
This page sets out what the AI/ML Evaluation Framework covers: its scenarios, the repositories it is built from, and how it relates to 5G-MAG's standards work.
What This Project Implements
The repository README describes the project as "the implementation of the AI/ML evaluation framework as defined in 3GPP SA4 TR26.927", used "for the evaluation of scenarios collected for the FS_AI4Media study". The evaluation testbed architectures, anchors and metrics are in 3GPP TR 26.847 V19.0.0 (clause 4 is titled "AI/ML Evaluation Testbed Architectures and Anchors"), whose reference [16] is this repository; TR 26.927 holds the use cases and the media service architecture. The README names TR 26.927; the clause that defines the framework is in TR 26.847. The README documents "common testbed architectures and anchors, metrics (e.g. AI/ML task metrics, feasibility/performance metrics), and specific details (such as test configuration and constraints) for each scenario evaluation".
The scenarios listed in the README are:
- Transmission of compressed AI/ML model data for automatic speech recognition
- Split inferencing for human pose estimation
- Split inferencing for object detection and labeling
- Split inferencing for hand gesture recognition
- Video super resolution in video call
- Federated learning
The repository holds datasets, models, evaluation scripts (one folder per scenario) and a Docker environment for running them. The README marks the project as under development; the hand gesture recognition and federated learning scenarios have no description in it yet.
The hands-on traffic characterisation work (network emulation and an AI traffic testbed) currently lives in the 6G AI Traffic Characterization Testbed reference implementation, which the AI/ML activity shares and builds on. See 6G AI Traffic Characterization Testbed for the testbed itself.
The traffic characterisation and network emulation tooling is not part of this repository's README. It is the 6G AI Traffic Characterization Testbed, which this activity shares, and that is the code you run today.
The scenario list above is the README's. TR 26.847 V19.0.0 clause 1 states its scope: "The present document provides details on feasibility evaluations for a given set of scenarios related to AI/ML in media services."
Key GitHub repositories
The AI/ML reference tooling is currently developed across the following 5G-MAG repositories. The live, auto-generated list (including auxiliary repositories) is on the Repositories & Releases page.
- 5G-MAG/6G-Testbed: network emulator and AI traffic characterisation testbed used for AI/media traffic evaluation
- 5G-MAG/Standards: tracking of 5G-MAG contributions to 3GPP, including AI/ML-related items
Some repositories may be private and under testing. Request early access.
Getting Started
Because the runnable code is the shared testbed, the quickest path is to start there:
- Read the 6G AI Traffic Characterization Testbed scope to understand the two building blocks (network emulator and AI traffic characterisation framework).
- Follow the Tutorials, which walk through the shared 6G AI Traffic Characterization Testbed tooling.
- Clone 5G-MAG/6G-Testbed and run a scenario against an emulated profile, then inspect the logged metrics.
Roadmap
General Project Roadmap
The Kanban board below tracks the planned and in-progress work for the AI/ML Evaluation Framework (benchmarks, datasets, and traffic-evaluation tooling). It is maintained live on GitHub.
Related
- 6G AI Traffic Characterization Testbed: the shared testbed this activity builds on
Refer to the rt-ai-ml-evaluation-framework repository on GitHub to use or contribute to this testbed. See How to Contribute for the process.