Why watching a movie is complicated..

TL;DR  : My videos sometimes looked like the first figure below until I found a setting to change, but there are still outstanding questions if you skip to the end.

Of course many of the geeks amongst you , having seen the title will realise immediately how little they know about MPEG , H.264 and and other alphabet soup which seems to fill the specification sheets of the equipment we use to just to entertain us.

I'm not going to talk too much here about about the specifics of Codecs because , I haven't read the appropriate standards , but am acutely aware I'd have to devote some serious time to actually understanding them. Instead this is a story about some very visible decoding artefacts I have been had the misfortune to find during mine and my partners watching of "Strictly Come dancing" from BBC HD. My partner being an avid fan means we watch it every week .

The artefact was a movement artefact only seen on left-right movement. Most noticeable if the was strong vertical line moving left-right across the screen , it became 'serrated'.

Let us start of with a little description about my setup, I am currently recording the Off-air stream from FreeSat (since when we moved in we had a dish and not an aerial) with MythTv. I then play the video with a separate settop box rather than using the mythfrontend.  All I use for that settop-box is XMBC on a RaspberryPi . The Raspberry PI is running RaspMc and set to track the release version. I've bought a license for the Broadcom codecs in the GPU so those should be in use doing the hard work of decoding the video stream. The final part of the chain is a Blaupunkt 42/131J-GB 42 LCD/LED TV.

I've been putting up with this artefact for some time, but finally decided a couple of weeks ago to do something about it. I'd  checked it the past and had seen the issue on my PC when the same recording was played back on the desktop using the players there - which should be an independent implementation since I'm using Broadcom's proprietary codecs on the Rpi. So having been fed up I raised an issue with the BBC Complaint site, since I haven't and still don't see issues on other BBC HD content (such as 'Doctor Who').

Unfortunately having raised the complaint it stuck me how long it had been since I actually did those test - and was unsure I could repeat them. So at the first opportunity I had I attempted to reproduce the fault using movie players on by desktop PC. This is where I hit a snag I found I couldn't be sure that I was seeing the same artefacts on the PC as on the TV in the living room. I went back to the living room to watch a clip of video which I new demonstrated the artefact to remind myself what it looked like.

Still showing the artefact.

 (See below for how it was generated)

This isn't actually a direct capture of the artefact but is a good almost perfect reproduction of what I saw if I paused the playback, - this is a still taken during a camera pan so is particularly full of of left-right movement.. But having seen it again I was sure that the artefact wasn't present on the decoder (ffmpeg) used on my desktop Pc to if at all and certainly to anything like the same extent as what I saw when watching the stream in the living room.

Unfortunately with modern TVs have their own image processing inside (they most do  - if only to scale image from differing resolutions to full the panel), and while watching it on a PC meant that the decoder could guess the best way to scale the image - TV are reknown about lying what the telling the image source about their actual resolution is (see Matthew Garrett's discussion on this ) .

As a result I needed to reproduce the fault on my living room display before I could really be sure what was going on. But I only had the Rpi under the TV to..., but looking into the telly cabinet from my admittedly recumbent position on the Sofa, I could see a new piece of equipment purchased which would play Streams out from MythTv (via DLNA)  - named my PS3 . (Yes newish, they are nice and cheap on Ebay on the PS4 is out. I don't play computer games , but its good for exactly this kind of verification.)

Watching this part of 'Take Two', clearly showed the fault was related to the Raspberry Pi. Good news, I could contact the BBC claim it was decoder problem at my end. And more importantly let may partner watch 'Strictly' at the coming weekend without those annoying serrated or jagged edges . I dropped an additional note to the BBC to this effect. By the way I'd like to comment here that having to provide updates through the same web form which collects an immense amount of ancillary data each time you update an issue sis a real pain. And also you can't add attachments so is practically impossible to so what faults you have.

By the time the weekend arrived , I had some more time to investigate ans it was nagging me again - I should at least try to raise a fault issue with Broadcom/Rpi Foundation about their codecs. Or possibly with the raspmc team if the issue was there. First off I wanted to capture a still like the one shown above which I could refer to in my bug reports. A bit of a google later, I came across this tool in github which promised to do the trick. I used it to capture this:-

Which looks alright. Well it's a bit blurry but. So something special is going on. Intrigued since I had it in the image viewer , I twiddled the zoom control a little, and suddenly there again was the artefact (this is how I actually captured the picture above it's a screenshot form the image viewer scaled to look as close the the original problem as possible), in fact I as zoomed in a and out the jagIn House Hospitality: Food for thought! Froged serrated edges moved across the image as well. Knowing a thing (but maybe not two) about images - "Aliasing" said I.

I also note that using GIMP as the image viewer didn't show this artefact - probably because with the correct algorithm is is possible to compensate for it . I 'll come back to this later.

You do however need some sort of high frequency pattern / oscillation to create possibility for this sort of aliasing , zooming into the picture thought it was easy to find. Every other line seemed was shifted the the left by approximately the same amount as the Jagged edge shown above. Given this was a still capture during a pan shot this is clearly characteristic of an interlaced video source.

So to recap, what we now know.

1. Strictly on FreeSat is transmitted in an interlaced frame format.
2. Something done in my RPi setup causes output aliasing.
3. The artefact was caused in the chain somewhere after where raspi2png captures it's image from .

I was a bit surprised to find that Strictly, which is after all one of BBC headline shows is transmitted with interlacing - acommunitity technique which I though was being slowly constrained to the dustbin of history. Indeed this article from 2004 shows the EBU (which the BBC are a member) clearly in favour of progressive (ie. non-interlaced video). According to wikipedia modern codecs like H.264 which the BBC use for Strictly show minimal (bandwidth) benefit for interlaced video . Certainly I can believe that given the minor peak I had into the encoding when trying to compare a 'Strictly' stream with a 'Doctor Who' stream. Of course a careful reading of that wikipedia page points to the existence of varying levels of decoder support for encoded streams.

But I also couldn't find anyway in the listing or the BBC website where they promise the HD content is 1080p (eg, FullHd, with no-interlacing) . But either way it's hardly a fault condition - regrettable but not a fault.

So that leaves the RPi, and RaspMC software I'm using on it, which is causing an aliasing artefact. Now for those paying attention , will have realised that to get that sort of aliasing some sort of (flawed) scaling operation must be involved somewhere  - like perhaps the scaling operation to add overscan to a video signal which a television might apply, but If that was the case why did I not see the same effect on the PS3?

While musing on that topic I had little memory of setting the 'overscan compensation' up on the RaspMc when I first installed the thing. I went back to that setting and found that if I tweaked it I could adjust the the jaggies in exactly the same was as zooming in an out So it was immediately clear to me that the effect was produced by the overscan compensation .

You see when you first turn on RaspMc it ask you to move a set of red markers around to define the edges of your screen. In other words there is you tell the system software how much of the 1080 frame is outside the display area of the TV. With this information the RaspMc system scales the image down so the whole 1080 frame is inside the viewable area. [And then presumably the TV scales it back up internally so it fits the Panel - but lets not worry about this. ]

It seems odd that raspi2png grab the screenshot before this ' setting' is applied, but that maybe a bug with the capture technique used. Further investigation showed that displaying the captured PNG with RaspMc faithfully reproduced the issues, so there are arguments both ways for this behaviour in raspi2png.

While investigating this I finally found an entirely new set of playback settings - which are only accessible during playback - which makes sense except during playback all the icons are invisible so you can watch the programme content (there is a way to make them appear , though).  And right in the middle of those settings was an option "Deinterlace" "On/Off/Auto".  It was set to 'Off' - and yes setting it to 'On' made the problem go away, and using Auto still seem to work.

Comparing other content on 'Off' and 'Auto' seemed no different, or indeed 'On' and 'Auto', which may suggest to me I haven't found the video which shows artefacts when de interlace is (erroneously?) attempted.

Where this leaves us

Despite having now found a perfectly reasonable solution to the problem at hand , I'm a number of questions were still nagging at me,

1. What was  the difference between the other BBC programmes I had watched and "Strictly".
2. Why did raspi2png capture the whole frame.
3. Why does the overscan compensation introduce scaling rather than use a proper filter?

Tackling these questions in reverse order -I'm pretty certain that the overscan compensation trick is done inside the GPU, the GPU also scales the image to produce standard definition output and all sorts, so I think it's all in this part of the set up - and it's more than likely that the just GPU doesn't have enough power to a proper filter for this function.
As to raspi2png behaviour it's pretty much a simple wrapper around one of the GPUs functions, it would be nice to see an option if possible to switch between the different frame buffers which are involved here.

Both of these two need to be taken to the Rpi community , so I'm going to head over there soon and ask around.

This leaves the first question , which part of which I should be able to investigate myself, after all there must be a program somewhere which analyses mpeg / avc files an describe the whay there are encoded - mustn't there. And indeed there was there is something called mpeginfo. Unfortunately all the BBC streams where describes as 'Interlaced: MBAFF' . MBAFF, as far as I can tell really doesn't mean a lot other than - "the content may be interlaced, but might not - but MBAFF is a really good setting let's use it".

I actually tried writing (ok, patching and compiling) my own version of mediainfo which would give more information but unfortunately it just made be aware of how complex Mpeg4 is... What I would be really interested if someone , perhaps at the BBC could explain why the interlacing issue  , and  esp. givin interlacing isn't exactly consider the highest quality option, is so much more evident on "Strictly Come dancing" compared to other BBC shows. Particularly since a Dancing show is going to be show lots of movement, you think it would be last option wanted.

UPDATE : I've just had a chance to go through some materials the BBC sent in response to my original query which I hadn't noticed, but states 1080i is there standard format for UK HDTV.