Video encoding, explained for DATV

Speaker notes

Chapter 2 followed the bits from the transport stream to the antenna. This chapter fills that stream: how a picture becomes bits, how the bits are packed, and how to check what the receiver hands back.

The goal at the end is a contest goal: send one still picture, your callsign card, as far as possible. Far means few bits: every halving of the bitrate lets the link run at half the symbol rate, and that is 3 dB.

Every pipeline in this chapter was run as shown, with GStreamer 1.28 and ffmpeg 8.0. The numbers on the charts are measured, not copied.

Speaker notes

This is the map for the chapter. Chapter 2 was the grey box on the right: the modulator and the demodulator. Everything to the left of it is this chapter.

The encoder turns pictures into a compressed stream: H.264 or H.265. The PES step gives every frame its timestamps. The mux cuts everything into packets of 188 bytes and adds the tables that tell a receiver what is in them: that is the transport stream, the TS.

The TS is the hand-over point in both directions. The modulator takes a TS, the demodulator gives one back. That makes it the place to look when something does not work.

Speaker notes

Raw: 720 × 576 pixels, 1.5 bytes per pixel in the 4:2:0 format every encoder uses (full detail in brightness, a quarter in colour), 8 bits, 25 frames a second: 124.4 Mbit/s.

440 kbit/s: 333 000 symbols × 1.32 useful bits per symbol for QPSK 2/3 with normal frames and no pilots (EN 302 307). That is 280 times less than raw video.

16 kbit/s is the channel of the last section: QPSK 1/4 at 33 ksym/s carries 0.49 useful bits per symbol, 16.18 kbit/s. That is 7 700 times less than raw. For moving video it is hopeless; for a still picture it is plenty, as we will see.

Speaker notes

H.264 (2003) and H.265 (2013) are both standards of ITU-T and ISO together. They describe the decoder exactly; the encoder is free, as long as a standard decoder can read what it makes. That is why x264 and a hardware encoder give different sizes for the same picture.

Speaker notes

The encoder works on blocks. For each block it first makes a guess: from the neighbouring blocks in the same picture (intra prediction), or from a decoded earlier picture, shifted by a motion vector (inter prediction). Only the difference with the guess goes on.

The transform turns that difference into frequencies, as the Fourier transform of chapter 1 does: a version of the DCT, on blocks of 4 × 4 to 32 × 32 pixels. A smooth block becomes a few low frequencies and many zeros.

The quantiser divides the frequencies by a step and rounds them. This is the only step that loses information, and the one you tune: the step size is set by QP, the quantisation parameter. Every 6 steps of QP double the step size.

The entropy coder (CABAC, an arithmetic coder) packs the result into as few bits as the statistics allow.

The loop below is a decoder inside the encoder. It predicts from what the receiver will have, not from the original; otherwise the small errors of quantising would add up frame after frame.

Speaker notes

An I-frame (intra) is coded on its own: a receiver can start there. A P-frame (predicted) only carries the difference with an earlier frame. A B-frame also looks at a later frame; it saves bits on moving video, costs delay, and gains nothing on a still picture, so we switch it off.

For a still picture the difference is zero. Measured: the I-frame of our callsign card is 4 958 bytes at QP 34, every P-frame 22 to 32 bytes: a header that says "every block is the same as before".

The catch is on the air. Every frame starts a new TS packet of 188 bytes, so a 23-byte P-frame still costs 188 bytes, and 25 of them a second cost 37.6 kbit/s for nothing. For a still picture the frame rate matters more than the codec. Section 04 comes back to this.

The distance between I-frames (the GOP, group of pictures) decides how long a receiver that tunes in waits for a picture. Here: an I-frame every 5 frames, at 1 frame a second.

Speaker notes

H.264 cuts every picture into macroblocks of 16 × 16 pixels. Each one carries its own header: its type, its prediction, its motion vector.

H.265 starts from coding tree units of up to 64 × 64 pixels and splits each one in four, again and again, only where the picture needs it. A plain sky is a few big blocks; an edge gets small ones. The drawing splits a block wherever it is not all one colour, down to 8 × 8.

Big blocks pay off in large, smooth pictures: HD video and up. In a small picture there are few plain areas of 64 × 64, so the gain shrinks. The measurements on the next slides show exactly that.

Speaker notes

The chain is the same; H.265 has more choice at every step, and the encoder spends more time choosing.

SAO (sample adaptive offset) is a second filter after deblocking that corrects small offsets in the decoded picture.

"Half the bitrate for the same quality" was the goal the standard was designed for, measured on HD video with people judging the result. On small pictures the gain is much smaller: the next slide.

For DATV: Longmynd, Ryde and the Portsdown receive both; an older set-top box may only know MPEG-2 or H.264. Check your receiver before the contest.

Speaker notes

Measured: one I-frame, encoded with x264enc and x265enc at speed-preset veryslow and a constant QP. Two pictures: a photo at 720 × 576, and a callsign card (white letters on blue) at 352 × 288.

The photo: from QP 34 to 40 the H.264 frame falls from 51 to 29 kB, the H.265 frame from 37 to 21 kB. H.265 saves 19 to 31 % at the same QP. Same QP is not exactly same quality between the two codecs, but it is close enough to show the trend.

The card: H.265 saves 11 % at QP 22, 2 % at QP 34, and from QP 38 it is larger than H.264. A small, plain picture leaves the big blocks of H.265 little to win.

The picture you send matters more than the codec: at QP 34 and 352 × 288 the photo is 20 kB, the card 5.5 kB. Plain colours and big letters are the cheapest picture there is.

Speaker notes

The left column is what you tune; the middle columns are the property names of the two GStreamer encoders. Picture size and frame rate are not encoder settings: they are set in the caps before the encoder, video/x-raw,width=352,height=288,framerate=1/1.

Constant quality (a fixed QP) gives every frame the same quality and lets the size follow. Constant bitrate does the opposite: it moves QP so the stream stays at the rate. For a still picture constant quality is simpler; the last section shows when constant bitrate helps.

The buffer (VBV) says how much one frame may overshoot the rate: x264enc takes it in milliseconds, x265 in kbit. A big I-frame needs a big buffer, and a big buffer means the receiver must wait longer.

Headers: a receiver that tunes in halfway needs the SPS and PPS (size, profile, settings) before it can decode anything. h264parse config-interval=-1 puts them before every I-frame; x265 does the same with repeat-headers.

Hardware encoders (vaapih264enc, nvh264enc, v4l2h264enc on a Raspberry Pi) have their own property names and fewer settings, and usually larger frames at low rates. For a still picture there is time enough for x264 or x265 on any computer.

Speaker notes

gst-launch-1.0 builds a pipeline from elements joined by "!". Data flows left to right. Between elements, caps (capabilities) fix the format: here 720 × 576 at 25 frames a second.

-e sends an end-of-stream when the source is done, so mp4mux can write its index; without it the MP4 file cannot be played.

The parser after the encoder (h264parse, h265parse) reads the stream and tells the next element what it is: profile, level, size. Muxers want it.

gst-inspect-1.0 x264enc lists every property of an element with its range and default. That is the manual to keep open.

Both pipelines were run as shown with GStreamer 1.28; the files are 2.0 and 1.1 Mbit/s.

Speaker notes

decodebin reads the PNG (or JPEG). imagefreeze repeats that one picture at the frame rate of the caps, num-buffers times, or for ever when it is left out.

videoconvert and videoscale bring the picture to the format and size of the caps: I420 is 4:2:0, the format of the raw-video slide.

mpegtsmux packs the stream into a transport stream. That is the next section.

On the receiving side, tsdemux unpacks the TS, h264parse finds the frames and avdec_h264 (gst-libav, the decoders of ffmpeg) decodes them.

Speaker notes

An encoder makes a bare stream of frames. A container adds what a player needs around it: which streams there are, when each frame is due, and how to find the frames.

MP4 keeps that in an index (the moov box), normally written when the file is closed. A player that misses the start, or a file that is cut off, has nothing to go on. That is why -e mattered on the previous slides.

The MPEG transport stream (ISO/IEC 13818-1, 1995) was made for broadcast. Everything is cut into packets of 188 bytes, and the tables that describe the streams are sent again every fraction of a second. A receiver can tune in at any moment, and a lost packet costs a piece of one frame, not the file.

DVB-S, DVB-S2 and DVB-T all carry it, and so do IPTV and HLS streaming. A TS made for DVB-S works over DVB-T unchanged; only the modulator and its bitrate differ.

Speaker notes

The encoder delivers NAL units (network abstraction layer): the SPS and PPS with the settings, and the slices with the coded picture. All NAL units of one frame together are an access unit.

The PES packet (packetised elementary stream) wraps one access unit with a header: the stream id (0xE0 for the first video stream) and two timestamps in a 90 kHz clock. DTS says when to decode the frame, PTS when to show it. Without B-frames they are the same.

The TS mux cuts the PES packet into pieces of 184 bytes and puts a 4-byte header in front of each. The first packet is marked as the start of a PES packet; the last is padded with stuffing in its adaptation field.

So every frame starts a new TS packet, and a frame of 23 bytes still costs 188.

Speaker notes

The header is the first thing to read when you debug. Three fields matter most, in blue.

TEI, the transport error indicator: the demodulator sets it when Reed-Solomon (DVB-S, DVB-T) or BCH (DVB-S2) could not repair the packet. Chapter 2's "knows when it fails" ends up here. A TEI count above zero means the link is at the cliff.

PID, the packet identifier: which stream the packet belongs to. 0 is the PAT, 0x11 the SDT, 0x1FFF null packets; the others are set by the PMT.

CC, the continuity counter: counts 0 to 15 per PID. A jump means a packet went missing: dropped by the receiver because it was bad, or lost on the network between the receiver and your computer.

The adaptation field, when present, carries the PCR (the clock) or stuffing.

Speaker notes

A receiver knows only one PID in advance: 0, the PAT (program association table). The PAT lists the programs and the PID of each one's PMT.

The PMT (program map table) lists the streams of the program, each with its type and PID: 0x1B is H.264, 0x24 H.265, 0x02 MPEG-2 video. It also names the PID that carries the clock.

The SDT (service description table, ETSI EN 300 468) gives the service a name. That is where your callsign goes, so the receiver shows it.

Null packets (PID 0x1FFF) carry nothing. They fill the stream up to the exact rate of the channel, and the receiver throws them away.

The PIDs here are ffmpeg's defaults, the muxer of section 04. GStreamer's mpegtsmux puts the PMT on 0x0020 and the video on 0x0041. The PIDs do not matter, as long as the tables say where things are.

Speaker notes

The receiver does not know how fast the sender's clock runs. The PCR (program clock reference), a 42-bit count of a 27 MHz clock in the adaptation field, tells it. The receiver locks its own clock to it.

Each frame carries a DTS and a PTS on that same clock. The decoder takes a frame from its buffer at its DTS and shows it at its PTS.

So every frame must be in the buffer before its DTS. On a fast channel that is easy. On a slow channel a big I-frame takes seconds to send, and the mux must send it early enough. Section 04 shows a muxer that does not.

ISO/IEC 13818-1 asks for a PCR at least every 100 ms. At 16 kbit/s, one packet takes 94 ms: almost every packet would carry a PCR. Software players (VLC, ffplay, the DATV receivers) cope with 500 ms; a set-top box may not.

Speaker notes

A modulator sends a fixed number of symbols per second, so it needs a fixed number of TS packets per second: the symbol rate times the useful bits per symbol.

DVB-S2, normal frames of 64 800 bits, QPSK, no pilots: a frame takes 32 400 + 90 symbols. QPSK 1/4 carries 16 008 − 80 = 15 928 useful bits in it: 0.490 bits per symbol, 16.18 kbit/s at 33 ksym/s. QPSK 2/3 carries 42 960: 1.322 bits per symbol, 440 kbit/s at 333 ksym/s. Short frames and pilots carry less; use the numbers your modulator software shows.

DVB-S: 2 bits × 1/2 × 188/204 = 0.92 bits per symbol, 30.4 kbit/s at 33 ksym/s (chapter 2).

DVB-T: the same TS. In 8 MHz with QPSK 1/2 and guard interval 1/32 it carries 6.03 Mbit/s; a narrower channel carries proportionally less.

When the video has less to say, the mux fills up with null packets. When it has more, packets queue up and arrive late: the bitrate of the TS must never exceed the channel.

Speaker notes

A QO-100 style stream: 720 × 576, 25 frames a second, H.264 at 350 kbit/s, in a TS at the 440 kbit/s of DVB-S2 QPSK 2/3 at 333 ksym/s, sent over UDP to the modulator. Use the address and port your modulator listens on.

is-live=true makes imagefreeze run at real time; without it the pipeline runs as fast as the computer can, which is right for a file and wrong for a transmitter.

The intervals are in ticks of the 90 kHz clock: 90000 is 1 s. The defaults are 9000 (100 ms) for PAT and PMT and 3600 (40 ms) for the PCR.

mpegtsmux has no setting for a service name: its TS has no SDT with your callsign. ffmpeg's muxer has one; section 04 uses it.

Measured with these settings, a test pattern for the card and a file for UDP: constant 440 kbit/s, every frame on time. With a detailed photo the I-frame (34 kB) arrived 0.7 s after its decode time: mpegtsmux starts its clock at the first frame and leaves no room for a big one. Section 04 has the way around it.

Speaker notes

When the picture does not come, the question is where it got lost: the radio link, the stream, or the decoder. The TS between demodulator and demux answers that.

Speaker notes

1. Lock. Longmynd (behind the MiniTiouner, and inside Ryde and the Portsdown) shows whether it has found the carrier, the symbol rate, the code rate and the MER. No lock means a radio problem: frequency, symbol rate, signal. Chapter 2 is the place for that.

2. Record. Longmynd sends the TS over UDP or into a pipe; a Linux DVB card or USB stick gives it on /dev/dvb/adapter0/dvr0. Record it to a file, so you can look as often as you like.

3. Contents. Without a PAT and PMT the receiver does not know what to decode. With an SDT it shows your callsign.

4. Errors. TEI: packets the error correction could not repair, a weak signal. Continuity errors without TEI: packets lost on the way to the computer, often UDP. PCR: too far apart, or jumping.

5. Decode. If the stream is whole and the picture still does not come, it is the video itself: no SPS and PPS before the I-frame, a codec or profile the decoder does not support, or frames that arrive too late.

Speaker notes

TSDuck (tsduck.io) is the toolbox for transport streams, free and open source. tsp is its pipeline: input plugin, processing plugins, output plugin. -I ip 1234 listens on UDP port 1234: use the port your receiver sends to. tsanalyze capture.ts gives a full report of a recording: every PID, every table, the bitrates, the errors.

On a Linux DVB card, dvbv5-zap tunes and -r sends the stream to dvr0; -P passes all PIDs, not just the program.

ffprobe shows the program, the stream type, the picture size and the service name from the SDT.

tools/ts-check.py is in this folder: plain Python, no packages. It counts packets, bitrate and continuity errors per PID, the TEI flags and the PCR spacing. Given the TS bitrate, it also checks whether every frame arrives before its decode time, the problem of section 04.

ffmpeg -v error with -f null decodes everything and writes nothing; every line it prints is a decoding error.

Speaker notes

Read from the top: each line only makes sense when the ones above it are fine.

TEI and continuity together tell where packets get lost. TEI with continuity errors: the radio link. Continuity errors without TEI: between the receiver and the computer, often UDP on a busy network or Wi-Fi.

The PCR line: the standard asks 100 ms at most. At 16 kbit/s that costs most of the channel, so low-rate DATV uses 500 ms, which software players accept. Test your own receiver.

"Too late": VLC, for one, writes "picture is too late to be displayed" in its log and drops the frame. With a still picture you may see nothing at all, while every other check is green.

Speaker notes

At the same transmitter power, the receiver collects noise over the bandwidth of the signal. Half the symbol rate is half the bandwidth: half the noise, so 3 dB more signal-to-noise ratio.

From 333 to 33 ksym/s is 10 times less bandwidth: 10 dB.

In free space, the signal falls with the square of the distance: 6 dB for every doubling. 10 dB is 3.2 times the distance. Over the horizon, through trees and over hills, the gain in distance is smaller, but the 10 dB is still there.

And the most robust mode: DVB-S2 QPSK 1/4 decodes at an Es/N0 of −2.35 dB, below the noise (EN 302 307, ideal receiver; a real one needs 1 to 2 dB more). It carries 0.49 useful bits per symbol: at 33 ksym/s, 16 kbit/s. The question for the rest of this chapter: does a picture fit in that?

Speaker notes

All four bars are the same callsign card, H.264 at QP 34, an I-frame every 5 seconds, measured with tools/ts-check.py.

With the defaults, mpegtsmux sends a PAT and a PMT every 100 ms. Each is a whole packet: 2 × 10 × 188 × 8 = 30 kbit/s, 27 measured. 64 % of the stream is tables; the picture is 15 kbit/s.

Tables once a second: 17.9 kbit/s, still too much for the channel.

1 frame a second instead of 5: the empty P-frames go from 5 packets a second to 1. 11.9 kbit/s: it fits.

The last bar is the pipeline of the next slides, with an SDT and padded with null packets to exactly 16.18 kbit/s: 12.8 kbit/s of content, 3.4 of padding. That padding is margin for a more detailed picture.

Speaker notes

Both streams: the same card, 1 frame a second, an I-frame every 5, at 16 kbit/s. Each bar is a frame on the air; the triangle below is its decode time. A frame must be on the air before its triangle.

Top, GStreamer's mpegtsmux with bitrate=16000: it starts the clock at the first frame. The I-frame then takes 3.5 s on the air, while it is due at once. It arrives 3.5 s late, and the P-frames behind it are late too. Measured over a minute: 48 of 60 frames late. Its latency property did not change that, nor did a live source.

Bottom, ffmpeg's muxer with -muxrate 16000 -muxdelay 4: the decode times are 4 s after the clock, so the I-frame (3.2 s on the air) is in time, with 0.5 s to spare. Every frame on time.

Rule of thumb: the mux delay must be longer than the time an I-frame takes on the air: its size × 8 / the TS rate. 5 kB at 16 kbit/s: 2.5 s, plus the tables in between.

The cost: a receiver shows the picture 4 s after it was sent. For a still picture that does not matter.

Speaker notes

This is tools/still-to-ts.sh, written out. GStreamer makes the H.264 stream and packs it in a plain TS on its standard output (fdsink); ffmpeg reads that, copies the video without re-encoding (-c copy) and muxes it again at a constant rate.

-muxrate is the useful bitrate of the channel, in bit/s. -muxdelay 4 sends every frame 4 s ahead of its decode time. -pat_period 1 sends PAT and PMT every second, -sdt_period 5 the SDT every 5 s, -pcr_period 500 the PCR every 500 ms.

-probesize and -analyzeduration make ffmpeg start after about a second of stream; with its defaults it waits for megabytes, which at this rate never come. The first packet still leaves about 15 s after the start: the encoder, the probe and the mux delay together.

pkt_size=1316 puts 7 TS packets in each UDP datagram, the usual size. Use the address and port your modulator listens on, or a file name to record.

Measured over 70 s: constant 16.18 kbit/s, PAT and PMT 1.5 kbit/s each, SDT 0.3, video 9.5, null packets 3.4; no continuity errors; every frame between 0.56 and 3.9 s early. The stream decodes with ffmpeg and with GStreamer.

For H.265: CODEC=h265 tools/still-to-ts.sh. Measured: the card's video was 11.6 kbit/s instead of 9.5, so on this picture H.264 is the better choice.

Speaker notes

A different trade. Constant bitrate with a buffer of 1 s (x264enc bitrate=10 vbv-buf-capacity=1000) caps every frame at about what the channel carries in a second. The I-frame comes out coarse; every P-frame after it spends its share of bits on the detail still missing, because the picture does not move.

Measured at 2 frames a second, an I-frame every 10 s: the quality climbs from 18 to 34 dB PSNR over 10 s, then the next I-frame starts it again. The video is 8.4 kbit/s.

The gain: no frame is big, so no frame needs seconds on the air, and even with mpegtsmux no frame was more than 0.7 s late. The cost: a coarse picture for the first seconds after each I-frame.

PSNR (peak signal-to-noise ratio) compares the decoded picture with the original: around 30 dB a picture looks good, below 20 dB blocky.

Speaker notes

The settings of tools/still-to-ts.sh, which are the settings measured on the previous slides.

Picture: design the card for the codec. Flat colour, no gradients, no photo behind the text, no noise. Measured at 352 × 288 and QP 34: card 5.5 kB, photo 20 kB.

Every value is a trade. An I-frame every 10 frames halves its cost but makes a receiver that tunes in wait up to 10 s. A lower QP sharpens the picture and lengthens the I-frame on the air: then raise the mux delay too.

With more channel (DVB-S QPSK 1/2 at 33 ksym/s gives 30.4 kbit/s) the same picture fits at a lower QP or a larger size.

Speaker notes

Everything in this chapter was measured on files and UDP, not on the air: no modulator and no receiver were in the loop. A real receiver can have its own limits.

Your receiver: Longmynd with VLC or ffplay accepts most of this. A set-top box may want a PCR every 100 ms, 25 frames a second, or no H.265. Find out before the contest, not during it.

Your own stream: modulator into an attenuator into your receiver, record the TS there, and run the five checks of section 03. The stream that leaves your computer and the one the receiver gives back should be the same, packet for packet.

The margin: lower the signal until the picture goes, and note the MER where it happens. That is your cliff; in the field you know how far you are from it.