Wednesday, November 04, 2015

About the HP IP KVM 286598-001

There seems to be bollocks all information about this on the HP site. So here is a brain dump of what I've got so far.

So first off - the serial console pinout is


So a straight forward 1-1 connection cable seems to work . The online manuals suggest 9600 8N1 , many of the setting have to be set through the serial console.

Print-screen -key  Brings the OSD up.
The Individual ps/2 adaptor don't communicate unless they have power in there keyboard plug, I'd expect similar form the USB ones but I haven't got one to try.

According to a forum post, if there is a serial console password you have to return it to HP to be reset at a charge but this is service HP no longer offer.

Oh guess, what yes - the one I bought of ebay for £30 (which admittedly is a real steal) has a password set.

Anybody a common password generator I could script a cracker from?

Tuesday, September 01, 2015

Another visualization trick - or annoucing timeline a new HTML5 app for timelines.

When writing freeform games, when tend end up with a lot of prose, mostly distinct that says things "A few weeks after that you received a letter".

And another character (eg on a different document) will have some text like ' 4 months ago you sent  a letter to.."

With all these relative dates, often relative to different event, so of those are relative as well, it can be very difficult to keep track of what you said happened in what order.

To ensure I was as consistent , I wanted to chart these out. In the first place I found the lovely vis.js visualisation library for the browser environment which has a timeline widget.

It was a simple matter to invoke the timeline widget with an array of events created from  by date arithemtic, and plot them. The vis.js gave me zooming , dragging etc so I could move about my timeline.

It showed me something I should have known was wrong and another, and something else a bit odd. So a success  - it has allowed me to view the timeline clearly. So successful that I had a request to use from someone who has never coded javascript before - and they managing to use it just about Ok .

Unfortunately this fact that version of the tool was so simple lead me to thinking - it can't be too hard to put a proper UI on this can it. It a bank holiday, I'll knock it up an Saturday have and the rest of the weekend to do the other things I have planned.

Famous last words , of course. By the end of Saturday - I had got it to remember the the links and allow the dependent to all about prettily when dragged, and by the end of Snday you could create events, rename events. And save your work. You couldn't load it back though.

It took Monday morning to get loading, and a good chunk  of the afternoon to get dependent object creation. Leaving me Monday evening to all my bank holiday chores.

Yeah - right.

So what you can see here is very much a work in progress -although I do have other things to be getting on with. My aim was something that can be a single file (+ some js downloaded from a cdn) people can run in their browser to manage these sort of dataset privately. It has moved a little way from that if with the addition of jquery and jquery-ui. But perhaps not to far.

At the moment I have the Javascript in seperate files, but perhaps I can a stage to build process to bring it together... but there are other feature it needs more...

Sunday, May 17, 2015

Visualization of ideas during brainstorming the easy (for a geek) way

At peaky, the cottages all have a modern LCD telly, while I didn't check this year, I'd be surprised if they didn't have a HDMI input .

This could be used so a laptop user could show some sort of ouT of visualization or similar to help during the early stages of game development - for myself I like to see the connections between character visually and for a number of years have been using the graphviz tools for this and similar tasks.

In fact I've been experimenting drawing these graphs for the game we wrote at peaky this year. Firstly I was surprised how easy it was, you can easily create a graphviz file - and it are relatively self explanatory.

You need a text editor, and be to happy using one, and there is a little bit of header and footer for the file.

So open notepad an type: (or use an online website such as or )

graph  {


And that all you need as a header an footer for an empty diagram (technically they are called graphs but I'm ignoring that for now ).

Then you literally just need to write in each connection like this:-

So if Alice shares plot with Bob we just write :-

Alice -- Bob

We have created a diagram with a line between Alice and Bob, it that simple so we can now add some bad guys.

Eve -- Trent
Bob -- Trent
Alice -- Trent

Put this together we would end up with a file like

graph {

Alice -- Bob
Eve -- Trent
Bob -- Trent
Alice -- Trent


The indentations not important , but it can make the file easier to read, and is second nature to most geeks to add it like this.

This produces a graph like so:

And this is where I found a few extra tools to make these super-easy on linux, but I'd be surprised if there weren't equivalent windows tools, or you can use the web tools above which do the whole lot automatically.

The first tool is graphviz itself, if you doing this directly on your computer you need to ensure it is installed. It adds a command called 'circo' which arranges the characters in a circle , and draws the links between them. There a couple of different programs included with graphviz which place the characters according to different
rules - but I've found circo works best for character links in games.

cicro and the other graphviz tools will output many different image formats, including SVG , PNG and JPG - to name the ones I find most useful.

I needed one final piece to put all his together in a manner which made the process simple for use during group plot development. I needed
way to trigger the image file to be regenerating from the dot file whenever it is updated . There a number routine which I found for this
entr for linux or when_changed and folder monitor for windows.

Since I use linux as a matter of course, entr was the utility I needed. Running the follow rather convoluted command then put into place a
few programs which would keet the image file updated.

ls | 
entr  bash -c 'circo ./ -Tpng > character_connection_new.png '

Since I had already established eog would update the screen with these pieces in place I can open the image view full screen on the tv (secondary display) and edit the text file adding and removing connections. Every time I save the file the diagram updates on the screen automatically.

My aim is that given that it is a simple line to add a connection between characters, this can be added during a brainstorming sessions for immediately visible to everybody involved.

Here is a fuller example from the game we wrote at peaky, I've made some minor changes and removed the actual character names to prevent spoilers, (the game won't be run in the form shown here ) .

From looking at this I can see that J is really no involved with much of the game, and I'd be worried about H,D,K and especially G not having enough to do. The other thing here is I've made use of some additional; features allowing me to colour the nodes. Links etc, can also be given colour.

Sunday, December 14, 2014

Another Mystery Machine update

I haven't spent much time on Mystery Machine recently, I've been working on a secret project which I'm either going to give up on, or maybe not. The secret project is going slower than I would like hence the indecision.

Anyway, I've taken a break from the secret project for all sorts of reasons - some personal - others not. At a recent event I realised I needed to use MysteryMachine but wanted to be sure I could interact with other who may not want to.

I came up with use a simple XML style export/import feature, but hit some issues due to some woolly think on the internal  (and external) APIs back in the early days.

The changes required are quite minor. But in checking what I had advertised as API I found the  documentation was not in as a good state as I would like . So I've fixed that.

I've finally three years after foreshadowing the move (!) taken down the trac site and main the bitbucket site the primary site for the wiki and the bug tracker. This was easier to do no that three year ago as Bitbucket now support Restructured text as wiki entries which is what All my original wiki pages were written in. I've also attempted to update the wiki to show the current status of the project better. In particular the developer information page has been updated, alongside which I've run pdoc to automatically generate some API documentation linked to from the developer page.

Now below when I make the announcement of the coming API those of you who are interested at least have the context to understand without to much digging.

The API Change

Which is an anti-climax given how much introduction it has . From future versions, objects EnumCategories and EnumAttributes methods *should* return system attributes.

Thursday, October 16, 2014

Why watching a movie is complicated..

TL;DR  : My videos sometimes looked like the first figure below until I found a setting to change, but there are still outstanding questions if you skip to the end.

Of course many of the geeks amongst you , having seen the title will realise immediately how little they know about MPEG , H.264 and and other alphabet soup which seems to fill the specification sheets of the equipment we use to just to entertain us.

I'm not going to talk too much here about about the specifics of Codecs because , I haven't read the appropriate standards , but am acutely aware I'd have to devote some serious time to actually understanding them. Instead this is a story about some very visible decoding artefacts I have been had the misfortune to find during mine and my partners watching of "Strictly Come dancing" from BBC HD. My partner being an avid fan means we watch it every week .

The artefact was a movement artefact only seen on left-right movement. Most noticeable if the was strong vertical line moving left-right across the screen , it became 'serrated'.

Let us start of with a little description about my setup, I am currently recording the Off-air stream from FreeSat (since when we moved in we had a dish and not an aerial) with MythTv. I then play the video with a separate settop box rather than using the mythfrontend.  All I use for that settop-box is XMBC on a RaspberryPi . The Raspberry PI is running RaspMc and set to track the release version. I've bought a license for the Broadcom codecs in the GPU so those should be in use doing the hard work of decoding the video stream. The final part of the chain is a Blaupunkt 42/131J-GB 42 LCD/LED TV.

I've been putting up with this artefact for some time, but finally decided a couple of weeks ago to do something about it. I'd  checked it the past and had seen the issue on my PC when the same recording was played back on the desktop using the players there - which should be an independent implementation since I'm using Broadcom's proprietary codecs on the Rpi. So having been fed up I raised an issue with the BBC Complaint site, since I haven't and still don't see issues on other BBC HD content (such as 'Doctor Who').

Unfortunately having raised the complaint it stuck me how long it had been since I actually did those test - and was unsure I could repeat them. So at the first opportunity I had I attempted to reproduce the fault using movie players on by desktop PC. This is where I hit a snag I found I couldn't be sure that I was seeing the same artefacts on the PC as on the TV in the living room. I went back to the living room to watch a clip of video which I new demonstrated the artefact to remind myself what it looked like.

Still showing the artefact.
 (See below for how it was generated)

This isn't actually a direct capture of the artefact but is a good almost perfect reproduction of what I saw if I paused the playback, - this is a still taken during a camera pan so is particularly full of of left-right movement.. But having seen it again I was sure that the artefact wasn't present on the decoder (ffmpeg) used on my desktop Pc to if at all and certainly to anything like the same extent as what I saw when watching the stream in the living room.

Unfortunately with modern TVs have their own image processing inside (they most do  - if only to scale image from differing resolutions to full the panel), and while watching it on a PC meant that the decoder could guess the best way to scale the image - TV are reknown about lying what the telling the image source about their actual resolution is (see Matthew Garrett's discussion on this ) .

As a result I needed to reproduce the fault on my living room display before I could really be sure what was going on. But I only had the Rpi under the TV to..., but looking into the telly cabinet from my admittedly recumbent position on the Sofa, I could see a new piece of equipment purchased which would play Streams out from MythTv (via DLNA)  - named my PS3 . (Yes newish, they are nice and cheap on Ebay on the PS4 is out. I don't play computer games , but its good for exactly this kind of verification.)

Watching this part of 'Take Two', clearly showed the fault was related to the Raspberry Pi. Good news, I could contact the BBC claim it was decoder problem at my end. And more importantly let may partner watch 'Strictly' at the coming weekend without those annoying serrated or jagged edges . I dropped an additional note to the BBC to this effect. By the way I'd like to comment here that having to provide updates through the same web form which collects an immense amount of ancillary data each time you update an issue sis a real pain. And also you can't add attachments so is practically impossible to so what faults you have.

By the time the weekend arrived , I had some more time to investigate ans it was nagging me again - I should at least try to raise a fault issue with Broadcom/Rpi Foundation about their codecs. Or possibly with the raspmc team if the issue was there. First off I wanted to capture a still like the one shown above which I could refer to in my bug reports. A bit of a google later, I came across this tool in github which promised to do the trick. I used it to capture this:-

Which looks alright. Well it's a bit blurry but. So something special is going on. Intrigued since I had it in the image viewer , I twiddled the zoom control a little, and suddenly there again was the artefact (this is how I actually captured the picture above it's a screenshot form the image viewer scaled to look as close the the original problem as possible), in fact I as zoomed in a and out the jagIn House Hospitality: Food for thought! Froged serrated edges moved across the image as well. Knowing a thing (but maybe not two) about images - "Aliasing" said I.

I also note that using GIMP as the image viewer didn't show this artefact - probably because with the correct algorithm is is possible to compensate for it . I 'll come back to this later.

You do however need some sort of high frequency pattern / oscillation to create possibility for this sort of aliasing , zooming into the picture thought it was easy to find. Every other line seemed was shifted the the left by approximately the same amount as the Jagged edge shown above. Given this was a still capture during a pan shot this is clearly characteristic of an interlaced video source.

So to recap, what we now know.

  1. Strictly on FreeSat is transmitted in an interlaced frame format.
  2. Something done in my RPi setup causes output aliasing.
  3. The artefact was caused in the chain somewhere after where raspi2png captures it's image from .
I was a bit surprised to find that Strictly, which is after all one of BBC headline shows is transmitted with interlacing - acommunitity technique which I though was being slowly constrained to the dustbin of history. Indeed this article from 2004 shows the EBU (which the BBC are a member) clearly in favour of progressive (ie. non-interlaced video). According to wikipedia modern codecs like H.264 which the BBC use for Strictly show minimal (bandwidth) benefit for interlaced video . Certainly I can believe that given the minor peak I had into the encoding when trying to compare a 'Strictly' stream with a 'Doctor Who' stream. Of course a careful reading of that wikipedia page points to the existence of varying levels of decoder support for encoded streams.

But I also couldn't find anyway in the listing or the BBC website where they promise the HD content is 1080p (eg, FullHd, with no-interlacing) . But either way it's hardly a fault condition - regrettable but not a fault.

So that leaves the RPi, and RaspMC software I'm using on it, which is causing an aliasing artefact. Now for those paying attention , will have realised that to get that sort of aliasing some sort of (flawed) scaling operation must be involved somewhere  - like perhaps the scaling operation to add overscan to a video signal which a television might apply, but If that was the case why did I not see the same effect on the PS3?

While musing on that topic I had little memory of setting the 'overscan compensation' up on the RaspMc when I first installed the thing. I went back to that setting and found that if I tweaked it I could adjust the the jaggies in exactly the same was as zooming in an out So it was immediately clear to me that the effect was produced by the overscan compensation .

You see when you first turn on RaspMc it ask you to move a set of red markers around to define the edges of your screen. In other words there is you tell the system software how much of the 1080 frame is outside the display area of the TV. With this information the RaspMc system scales the image down so the whole 1080 frame is inside the viewable area. [And then presumably the TV scales it back up internally so it fits the Panel - but lets not worry about this. ]

It seems odd that raspi2png grab the screenshot before this ' setting' is applied, but that maybe a bug with the capture technique used. Further investigation showed that displaying the captured PNG with RaspMc faithfully reproduced the issues, so there are arguments both ways for this behaviour in raspi2png.

While investigating this I finally found an entirely new set of playback settings - which are only accessible during playback - which makes sense except during playback all the icons are invisible so you can watch the programme content (there is a way to make them appear , though).  And right in the middle of those settings was an option "Deinterlace" "On/Off/Auto".  It was set to 'Off' - and yes setting it to 'On' made the problem go away, and using Auto still seem to work.

Comparing other content on 'Off' and 'Auto' seemed no different, or indeed 'On' and 'Auto', which may suggest to me I haven't found the video which shows artefacts when de interlace is (erroneously?) attempted.

Where this leaves us

Despite having now found a perfectly reasonable solution to the problem at hand , I'm a number of questions were still nagging at me,

  1. What was  the difference between the other BBC programmes I had watched and "Strictly".
  2. Why did raspi2png capture the whole frame.
  3. Why does the overscan compensation introduce scaling rather than use a proper filter?

Tackling these questions in reverse order -I'm pretty certain that the overscan compensation trick is done inside the GPU, the GPU also scales the image to produce standard definition output and all sorts, so I think it's all in this part of the set up - and it's more than likely that the just GPU doesn't have enough power to a proper filter for this function.
As to raspi2png behaviour it's pretty much a simple wrapper around one of the GPUs functions, it would be nice to see an option if possible to switch between the different frame buffers which are involved here.

Both of these two need to be taken to the Rpi community , so I'm going to head over there soon and ask around.

This leaves the first question , which part of which I should be able to investigate myself, after all there must be a program somewhere which analyses mpeg / avc files an describe the whay there are encoded - mustn't there. And indeed there was there is something called mpeginfo. Unfortunately all the BBC streams where describes as 'Interlaced: MBAFF' . MBAFF, as far as I can tell really doesn't mean a lot other than - "the content may be interlaced, but might not - but MBAFF is a really good setting let's use it".

I actually tried writing (ok, patching and compiling) my own version of mediainfo which would give more information but unfortunately it just made be aware of how complex Mpeg4 is... What I would be really interested if someone , perhaps at the BBC could explain why the interlacing issue  , and  esp. givin interlacing isn't exactly consider the highest quality option, is so much more evident on "Strictly Come dancing" compared to other BBC shows. Particularly since a Dancing show is going to be show lots of movement, you think it would be last option wanted.

UPDATE : I've just had a chance to go through some materials the BBC sent in response to my original query which I hadn't noticed, but states 1080i is there standard format for UK HDTV.

Sunday, June 22, 2014

python_autoprofile - A simple way to profile all python scripts on you run.

I've just created this package over at github.

It really only useful for debian users, but other may be able to do something similar , but I don't know
what features rpm has which replicate dpkg-divert.

What this package does is allow you to add extra command line switches as 'default' to the invocation of an python script on your system. It takes these extra parameters from the environment, so unless you set any
of the it special Envvars it does nothing.

It has two special environment variables the first XPYADD_ARGS is literally just prepend to the the python command line every time python is run. I use this to add the '-3' switch in my development environment so I can worry about python3 compatibly during development . By  placing it in an environment variable is one of the simplest places to make production and development different without added processes etc, which might get forgotten. But not having '-3' in production it also means unnecessary warnings are in our production logs.

But I also use this package in production for the it's other feature the one it is named for. If the PROFILING environment variable is set, it may add "-m cProfile -o $UNIQUE_FILE" to the python command line. The will cause the python execution to be profiled and a stats file saved to UNIQUE_FILE.

I say may, because I don't want to slow production down to much by running this on every python invocation, so it uses a configuration file' /etc/python-autoprofile/PROFILING_FREQ' to decide randomly on each run whether to do so or not. The idea is that you can configure what proportion of scripts run slightly slower -as they have profiling enabled - so as not to disturb you users perceived performance but still get a broad brushstroke view of your python performance.


Sunday, February 09, 2014

Mystery Machine update.

I haven't been working very hard on this recently , but it isn't a dead project. I hit a problem a year ago which sent me spinning in another direction for a bit. I intended to blog about it separately as it is I believe a weakness in Python as a language when using in a parallel environment.

I found it because MysteryMachine uses threads for responsiveness in the case of blocking syscalls . I've spent a long time thinking about it and about a writing a PEP about the fix. The closest PEP is PEP419 , but I had limited success writing a blog article to capture my thoughts.

In the end a comment from Guido - about handling the issue differently - struck home and I realised I had all of that framework already, I just needed to make a whole lot of simple changes to MysteryMachine. I did and committed those changes a few weeks ago. But it doing so I discovered something else.

You see when I last worked on MysteryMachine my primary development environment was Debian Lenny (5.0) . Thing have moved n since than and I've upgraded my main machine not just to Wheezy (the current stable release  - version 7.0) but to the test release codenamed Jessie. This is relevant because the standard python2.5 setup for MysteryMachine is no longer part of the Operating system. I've managed to created virtualenvs for testing 2.5, 2.6, 2.7 just. The python 2.5 environment was a bit of challenge to setup.

As a result I'm putting user on notice that support for python2.5 will disappear at the next milestone release (probably 0.30). I don't intended even then to immediately remove the python2.5 support code - and there is code in there to specifically support the python2.5 environment - but python2.5 will be removed from my standard testsuite.