Ofcom carry out work to verify and measure mobile coverage, and make that available through outlets including their mobile coverage checker and regular Connected Nations reports.

Last year’s Digital Radio and Audio Review, published in April 2022, also made recommendations for work on mobile networks and radio streaming, specifically:

R32: Industry should work closely with Mobile Network Operators to promote the build out of robust mobile data networks (5G) and deliver on-demand, streamed listener services focused on in-car listening.

R33: Following on from the Plum report, radio broadcasters, transmission providers and Ofcom should initiate a programme of field-testing and trials to review and validate the Plum findings on 4G/5G coverage. The results of this testing should be discussed with Ofcom to ensure they include in their Connected Nations reporting a measure appropriate for reliable radio/audio streaming.

In Project Timbre, we are investigating these issues by focussing on the real-world Quality of Experience (QoE) for listeners using BBC Sounds. We are concentrating on live radio as the most challenging use case - after all, you can’t download something before it has happened live. However, the concepts we’re exploring can also apply to streaming of on-demand content.

Why we are interested?

We have a high confidence in the QoE delivered by our extensive network of traditional broadcast transmitters. These are downlink-only and dedicated to live audio delivery, being built for the specific requirements of broadcast.

Mobile networks, in contrast, are multi-purpose, highly complex and built to satisfy a broad range of competing requirements. These bi-directional IP networks allow the full range of BBC content to be made available, and offer the potential for new experiences and personalisation. Here, it’s harder to have confidence in the QoE since it depends on many factors, not least the intricate interaction between the network performance and the reactions of the algorithms in the playback client to dynamic changes in network characteristics.

However, this bi-directional nature of the mobile networks give us the opportunity to collect feedback on performance. We can then use this data to:

• try to understand the impact of predictions of mobile coverage on Quality of Experience; and
• optimise products like BBC Sounds to make them the best they can be in a mobile environment

What are we measuring?

We’ve built an augmented, private prototype of the Android version of BBC Sounds. Our engineers are using this internally to collect data on the real-world Quality of Experience (QoE) over mobile networks as they are today. We can explore variations due to location and time of day and use that data to identify the key metrics that have the greatest impact on QoE. It also enables us to have meaningful conversations with industry about how we can work collaboratively to make the QoE the best it can be and to examine the role that network features such as 5G Media Streaming could play. We’ve started with Android, since its APIs provide access to a richer set of low-level data on the mobile network than we have access to with iOS.

The image below shows what we’re doing:

The live radio stream originates at the BBC, where it is packaged as a sequence of MPEG-DASH audio segments. These are then delivered using HTTP via internet infrastructure, Content Delivery Networks (CDNs) and the mobile network to the BBC Sounds app running on a smartphone (or, in the future, a connected car). The sequence of discrete audio segments is then reassembled into a continuous stream of audio that is then played back to the listener.

We currently collate four types of data shown in the figure:

1. QoE metrics: These are derived from the audio playback client (the BBC’s Standard Media Player) and provide information about buffering instances (i.e. pauses in the audio playback) and how far behind live the listener is;
2. Audio Delivery metrics: These are provided by the underlying HTTP library, used to download the sequence of MPEG-DASH audio segments that constitute the live stream, including items such as the time taken to transfer the audio data and the time taken for the initial client request to be serviced (Time To First Byte, or TTFB);
3. Network Quality metrics: Using the Android Telephony APIs, we can collect detailed information about which mobile network is in use, its signal strength, signal quality and the primary serving cell ID;
4. CDN View: Unlike the above metrics that are all reported client-side, we can use logs from the CDNs to give us a server-side view of what is happening with the connection and the transfer of each audio segment.

This data is then regularly recorded and logged against location and time of day.

The client-side data (1, 2 and 3 above) is collected using MQTT messages that are delivered to a back-end Influx database. To avoid bias, we ensure we capture data from any areas with no mobile signal, with any messages that can’t be delivered being buffered in the client for delivery to the database once connectivity returns. We use a Grafana dashboard for real-time monitoring and debugging of the system.

In the example dashboard below, you can see the reporting of the playback state (e.g. PLAYING or BUFFERING), how far behind live the audio sits (‘smp_progress_behind_live_s’), the TTFB (‘Audio Segment Download Open Timer’) and mobile network metrics such as signal strengths (RSRP and RSSI) and cell handovers (‘cell_id_change_count’).

Quality of Experience and buffering

The audio segment duration on BBC Sounds is 6.4 seconds. To play uninterrupted, live audio and keep up with the live edge of the broadcast, the BBC Sounds app therefore needs to download successive MPEG-DASH audio segments on average every 6.4 seconds.

Setting aside any audio buffer on the device, failure to download any new segment within 6.4 seconds will firstly result in a pause in the audio. Secondly, the audio will resume where it left off when the pending segment downloads, albeit with the listener delayed further behind the live edge. The Quality of Experience will be impacted by both these effects.

Furthermore, the impact of this audio delay for the listener differs depending on the content being listened to, with live sport being potentially most critical – nobody wants to hear from their mates by text that an all-important goal has been scored before they hear it for themselves!

Under ‘normal’ conditions, where the entire distribution chain (including, for example, CDNs and mobile networks) performs sufficiently well, successive audio segments will typically arrive within a very short time of being requested (i.e. in the order of tens of milliseconds). Under such conditions, uninterrupted playback is guaranteed.

However, we know that playback interruptions (‘BUFFERING’ events) do happen in practice. Barring any erroneous operation of the BBC Sounds app itself – or being in an area without any mobile coverage – playback interruptions will be the result of excessive (i.e., more than 6.4 seconds) delay in receiving the requested audio segment data.

The presence of the audio buffer in the app decreases the chance of a delayed segment resulting in a playback interruption, and allows for a seamless switch between different modes of reception such as 4G to/from WiFi. There is a trade-off between the fill level of this buffer and the tolerance to excessive delay; a fuller buffer means that the client can wait longer for a segment to be delivered before an audio interruption occurs. However this resilience comes at the cost of a longer delay behind live for the listener compared with traditional broadcast reception.

An added complication is that this audio buffer will vary in how full it is, depending on how the user has controlled playback (e.g. pausing or fast forwarding). Crucially, the audio buffer level will also depend on any previous audio interruptions as a result of excessive delay that the client has been exposed to, such as having travelling through areas of poorer mobile coverage. The net effect is that different users experience different QoE depending on their past behaviour; indeed a single user may even experience a different QoE in the same location on the return vs. the outward leg of their journey.

In summary, the listener’s QoE is a complicated product of a number of factors, including user behaviour, exposure to previous network outages or delays and the behaviour of the playback client in reaction to those.

As a result, we're interested in occasions when the reception of the next segment is excessively delayed since this has the potential to cause an audio interruption. It can also act as a common currency, not impacted by variation in the fill level of the client audio buffer.

The impact of excessive delay can be seen in the particular example below. The graphs depict data from four different handsets, each connected to a different mobile network. In this case the handsets were in a static location, bringing into sharp relief the variation in QoE that can be observed by different users even in a single place.

The additional ‘BUF_4_SEEK’ state is used to distinguish between buffering that has occurred at the behest of the listener (i.e., when changing station, fast-forwarding, etc.) and buffering that results from the performance of the underlying mobile network.

As can be seen, over the course of the particular 30 minutes depicted here, several ‘BUFFERING’ instances occur, resulting in pauses in playback, due to the relevant audio segments not being delivered across the mobile network in time. The net effect of these is a spread of audio delays from around 7 to 24 seconds, across the four handsets.

It is interesting to note that, in this example, the measured signal strength (RSRP) appears generally acceptable throughout, although it varies significantly over time. However, the Time to First Byte (TTFB) can be significant and, in some instances, peaks at a time period at or above the length of the audio segment itself. In these instances, the actual audio data (which is only of the order of a few tens of kilobytes) is often transferred almost instantaneously, potentially giving an unrealistically high estimate of transfer speed if only the time taken to receive the payload is measured.

Digging deeper

The project is currently a small-scale trial, with around twenty handsets in the hands of engineers and volunteer members of staff; there are no plans to deploy this publicly. Despite this limited scale, we already have data for around 6,500 km of road and tens of millions of recorded data points. And while each individual measurement is a snapshot of the performance in a particular location at a particular time, we can use statistical techniques - of the sorts we’ve been using for many years to analyse broadcast networks - to build up a better picture of when and where the mobile networks work.

Statistical analysis and QoE metrics

To get more statistically significant results, we first need to recognise that streaming radio/audio is different to broadcast, in particular:

• Audio is segmented into chunks (of 6.4 seconds duration);
• The connection speed is variable rather than a constant bit rate;
• The mobile coverage for streaming of live radio does not appear to be solely determined by signal strength nor by raw transfer speed (since the TTFB can also be significant).

To develop Quality of Experience metrics, we aggregate our data points into 100m × 100m map squares (pixels) and are currently examining:

• The proportion of time the audio was interrupted due to buffering - with the proviso that this will be affected by the listener’s past behaviour (see above);
• The Time To First Byte (TTFB) - since this can often dominate over the transfer time of the actual audio data; and
• The probability of the availability of a given capacity (bit rate) within a map pixel – this would allow us to make a projection for the expected service coverage for a given required bit rate (e.g. audio or video).

This approach also allows us to start comparing our data with other publicly available data sources and statistics. If we can better understand and incorporate publicly available datasets into our work, we can increase our knowledge of how network performance affects BBC services. Additionally, we could encourage and explain the benefit of collecting and/or making available additional data that also encompasses the requirements of streaming services, for the benefit of the wider public and industry.

A couple of examples of our mapping into pixels are shown below. The shaded areas with a blue outline represent geographical areas which may be of interest and for which various statistics may be produced - for example, the percentage of roads, or rail that may experience various levels of network delay (i.e. TTFB, ‘Downlink Open’).

Conclusions

The data we’re collecting and analysing is giving us insight into real-world Quality of Experience and this certainly appears interesting. However, networks are complex and evolving - adding coverage, improving capacity and innovating with new architectures, capabilities and generations (4G, 5G, 6G, 7G …). Our data is therefore not the whole story.

Although we’ve only presented some snapshots of our data, they illustrate several points that we’re seeing more widely, in particular:

• The impact of the fill level of he client buffer means that the use of buffering instances - i.e. audio interruptions - as a QoE metric has to be done with care;
• A better understanding of the characteristics of mobile networks - in comparison to fixed networks - brings opportunities to optimise streaming audio apps for use on the go;
• Sufficient mobile signal strength (RSRP) is not a guarantee of coverage for a specific service (in this case, live audio streaming);
• Time to First Byte appears interesting in this context; and
• Transfer speeds need to be measured with care, particularly when dealing with audio segments that may be relatively small in size, and where the TTFB can dominate over the data transfer time;
• Not every listener to live streaming radio will experience an event at precisely the same time, and they will typically be somewhat behind listeners to our traditional broadcast radio networks - something our work on Low Latency Streaming is addressing. This could have an editorial impact that needs to considered.

There’s much more work to do in Project Timbre. We know, for example, that bottlenecks can and do occur at different points in the distribution chain; focussing solely on a client-side viewpoint doesn’t allow us to necessarily distinguish between these. We’re therefore currently looking in detail at the server-side CDN access logs mentioned above as a means to to see if we can start to unpick these issues. The intention is that we can then focus our research efforts on the relevant points of this chain.

As set out by the Digital Radio and Audio Review, we’re keen to work with industry and other broadcasters to better understand all these issues, and work across the distribution chain to optimise QoE and efficiency of delivery as well as explore how concepts of coverage and buffering might be better relayed to the public.

BBC R&D - 5G Media Streaming

BBC R&D - Low Latency Streaming