Navigating twisty little passages without any documentation

The windows API

2011-11-15T20:39:00.001Z

I'm not in general a fan the win32 api, which for those of you who know me well won't come as much of a surprise. But while working on a the bottom layer of the ACID complaint store stuff for MysteryMachine , I realised that I could just use TxF on versions of windows that are late enough. And indeed it seems stupid not to, so I decided to rewrite just the bottom layer of the transaction handler to use TxF when available.

Now for those who don't know , MysteryMachine is written in Python which is not the most natural language for direct access to the Win32 API which is primarily called from C/C++. There are some bindings also for C#, but I get the impression there is quite a lot between C# and the kernel when used this way.

So it became time for me to see how I could call this functions easily from Python. I looked into the Python toolbox and found ctypes which is perfect for the job. You can literally just mention a DLL by name, and construct a tuple of arguments. The arguments have to be made from special ctype objects which represent the types primitives available in C. You can read the documentation and the link above to find out more about ctypes itself.

Now I just wanted a simple wrapper around the TxF functions so I didn't have to go through all that rigmorole of type conversion each time, so it seemed that each function I wrote would be of the form of:-

def Windows32Function(*args,**kwargs):

type_list [ HANDLE, LONG , LONG, POINTER, LONG ,LONG, HANDLE, LPCWSTR ]
    out_args = []
    for arg,argtype in zip(args,type_list):
         out_args.append(argtype(arg))
  
    ctypes.dll.Windows32Function(*out_args)

Now the think I really don't like about the Windows Api is the large number of arguments that windows functions regularly take , and which 99% of the time you can just you the same default option, and it struck me - Python has first class support for keyword arguments, that what the kwargs argument above is for.

So I came up with the following:-

def do_args(args,kwargs,name,dflt,factory = None):
    if args:
        dflt = args.pop(0)
    val = kwargs.get(name,dflt)
    if factory:
 val = factory(val)

    return val


def CreateFileTransacted(*args, **kwargs):
    args = list(args)
    lpFilename = do_args(args,kwargs,'filename',None,LPCTSTR)
    dwDesiredAccess = do_args(args,kwargs,'desired_access',const.GENERIC_READ,DWORD)
    dwShareMode = do_args(args,kwargs,'share_mode',0,DWORD)
......

Looking at the code above , I can use the do_args function to take the grunt work out of processing each argument. , You should be able see that do_args takes an keyword name, a default value and a type in that order, as well as the arguments lists from the original function.
You might be wondering about the first line of the CreateFileTransacted wrapper, it is important that we use a args in a list forms as the tuple we start with (as all tuples in python are) is immutable so we can't pop the arguments from it as we go.

It takes an argument out of the positional list, check the args name in the keyword list, (keyword overrides positional in this implementation) and if neither is there will use the provided default. Then finally it casts it into
the correct type for ctypes to use. Here I name the arguments before I call the DLL entrypoint but thats not necessary a list can be built up like the original version did.

What's really good about this is that do_args handle all of the marshalling ctype requires the use to do, so is quick to write and the code is simple enough to be easy to verify. But the really amazing thing about this is it makes the windows API much more pleasant to work with. I managed to get most of the TxF implementation included all the windows functions done in mainly a day. There has been lots of messing around since to check it multiple platforms and deal with the inconsistencies but it was much quick to code windows calls this way. There are two big contributors to this ease which I don't normally get with windows programming and the first was the simple interactive nature of Python , I could very simply load the wrapper modules into a python console session and start calling the Windows API directly.

Normally this would be a pain in the proverbial , as the number of arguments is a large and would each have to be specified in the right order and with the right number of intervening NULL arguments for the features unused, but because I could really on the wrapper providing me with all the sensible defaults I only needed to provide the one or two arguments I was actually using it became much quicker to find out exactly what the real behaviour of windows was. I actually found it felt like windows behaved for once - I was quite shocked.

However as I was to discover while TxF behaved fine, windows file semantics meant that I had more problems head, but I save that for a later post.

New toy...

2011-05-22T13:14:00.002+01:00

I have just received a Kindle, bought somewhat cheaply from ebay, and I'm looking forward to modifying the running software on the device to add an Ssh shell, and remove some of the reporting to Amazon features.

The simplest way of stopping (a non-3G one like mine) from reporting to Amazon is to non-connect it to a wireless network. But I'd quite like to get a terminal running on it and a ssh client so that I can use it as a display for some other tasks when sitting outside. Unfortunately this is a little pointless if I don't connect to a wireless network.

In the bright sunlight LCDs often don't have enough contrast to compete with the uncontrolled nuclear reaction in the sky - eInk being a reflective as opposed to backlit-transmissive display technology should cope much better. I'm a little concerned the display's speed won't really be up to this use , and that the keyboard will be fiddly - but I am willing to try . It the best way to actually find how usable it is after all.

In the mean time I've copied the novels and papers I already have in pdf and mobi format onto the device (via USB) and have spend some time using it as a simple reader. One problem with the display I have noticed is that due to the electrostatic nature of the display it tend so attract dust and hairs, (particularly cat ones) on the display surface. The surface is so immaculate-matt white normall these are much more distracting than on a LCD/CRT . The other reason for this is while it is easier on the eye as the image forms on the surface of the display (CRT/LCD form the image just inside the display) so any dirt that falls on the display is in focus as you are reading the kindle.

On a side note about those looking for MysteryMachine updates, I have written a transaction manager for a 2PL locking scheme , which does the hard work for the ACI of ACID compliance. For the 'D' (Durability) I need to augment the store API as the store needs simple atomic write/undo/checkpoint type features. This is easy to implement on a store backend which already supports transactions , but harder otherwise. I'm also worried about the performance impact this might have - the store is already the slowest part of the system.

Until I've worked out the store API there is little point in commiting what I've got since it isn't invoked by anything yet. The Transaction manager is a per-system object so in the future alternate transactions managers might be possible (a 2V/TWR oneloks like promising idea.)

MysteryMachine Update...

2011-04-10T13:04:00.002+01:00

2Version 0.14 is tagged in mercurial, I need to create the distribution archives and upload it.

This was going to happen this weekend, but I left my laptop which is my primary development machine for this project in thne office on Friday. I've been working on my partner's 'spare' laptop an EeePc 901. Which after some heroic work with aptitude will run the required software - it's not fastest , or most comfortable machine to use - if nothing else just due to it's small form factor.

I'll update the trac site with a copy of the zip and tgz soon, meaning the early part of next week if all goes well.

I still haven't decided on what to do about project hosting. Ho Hmm.

I have also discovered a serious bug involving inheritance and list attributes, there is a unit test for it but that has been passing becasue of a bug in dict_store (fixed in my working copy), this looks like its going to be an awkward one and might trigger the complete rewrite of the Attribute update path and store API which ahas been in the back of my mind for a while. If it does look forward to the something like ACID compliance at that level. I will probably add some sort of transaction support too.

This for the average user this means we can guarantee better predictability of behaviour when you ask the system to do things it can't do, it wont corrupt your data. And should improve, make simple undo paths easier to code.

2011-03-26T12:55:00.004Z

It strikes me I haven't posted about the state of MysteryMachine for a while, in fact well over a year.

Well during that year good things have been happening , which if you have been paying attention to the repository history you might have seen.

First off all during the months of September and October I ported the Siege Mentality game written by J. Tuomas Harviainen for use as an example game for mysterymachine into the MysteryMachine database. I then ran the game at Consequences using only MysteryMachine to produce player character sheets.

Some more work is need to streamline this operation but actually using the program in anger at this stage was a very good thing to do, so I could see the what are the most important missing features are.

Indeed I wouldn't have been able to even do that without one import last minute feature, The Bidirection reference. This important feature is a special new attribute type which allows two mysterymachine object to be linked together - to make this work a bidilink - as it is known, uses another new feature - the two stage commit - to keep a pair of attributes in sync. This attribute is needed because each end of the link is modelling by its own attribute, this means each end of the link has a handle which can use to follow the link in the opposite direction, while allowing the code to ensure if one end of the link moves the other immediately reflects that change. A link which is moved forces the other end to point to the None object to indicate that is is currently unconnected.

Unconnected link attributes such as this are known as anchorpoints, and a link can only be connected to another anchorpoint. This is because an anchorpoint is not just an empty node, but it also stores which of it's ancestors in the database any link which partners with it will point to.

To give an example a character object may have a list of anchorpoints called 'plotroles' , each of these anchorpoints are places which you could connect a link from the plotrole to the character which takes that role in the plot. The anchorpoint in the character object knows that when the link from plotrole is followed that the follower wants to end up on the character, not the anchorpoint , or the list.

The module which implements this currrently has some pretty good examples in it's documentation , and I'll put them on the MysteryMahcine wiki which is also overdue an update.

The best news is there is now a UI. This means it can start to be used by ordinary people rather than programmers. Up till now you have only been able to manipulate MysteryMachine games and system via actually exceuting python code, but now there is a UI which should work on Mac,Linux & Windows - although I have currrently only tested on linux.

There is a plan for myself and my partner to write a full game in in time for November but we shall have to see how that goes.

Trac or bitbucket.

2011-03-26T12:20:00.004Z

Updated: 12:55 : I was attempting to blog from my andriod phone , but it posted it live rather than to drafts.
I've noticed that a recent server upgrade has broken the Trac install used by mysterymachine.
Trac was never very fast or responsive on my server and has no account creation support as standard (it is available as an extension). All accounts needed to be created by the site administrator.
This puts a roadblock in the way of causal users trying to report bugs. Not that I've hand any yet but I want to be able let all comers report bugs.
For sometime I've also had a mysterymachine archive on bitbucket, since bitbucket also provides a wiki and bug tracker as well as a large community. I'm strongly tempted to make that the new primarily for MysteryMachine. If I do so I will migrate the existing wiki content there. I will still host a copy of the MysteryMachine repo at hg.backslashat.org , so it is just the the issue tracker and the wiki which are really moving. Does anyone have any opinions.

Another Milestone

2010-03-18T21:51:00.003Z

Earlier today I committed changeset 'ce4f0384c0b329ee85b08b40ee4bd6087f18fb00' to the MysteryMachine repository With the addition of this code the system can now actually be used to generate character sheets and other report documents. There still isn't though a UI, an installer or a decent plugin manager and even the database isn't entirely feature complete. The most pressing of the above is the installer so that got to be my next push. I;m probably going to take a short break , I've got some creative writing for a friend coming up - and unfortunately he has a database with all the background information in which needs fixing first too.

Find and discover switches, well the smart ones.

2010-02-17T12:22:00.009Z

Like many people I have occasionally forgotten small details when under pressure, and even worse sometime forgotten to write them down. Added to the fact I regularly visit or remotely handle to handle sites setup by other people I don't always know the configuration of a site switches.

Most modern switches have a web interface or at least a telnet one - but serial ports are becoming less common - even if there are connected.

Cisco devices (and some Linksys now), will reveal their presence to you via CDP, just sit on the net with your favourite sniffer running and watch for CDP packets. However Netgear switches don't do this. Netgear do offer a downloadable discovery program - but it is for windows only, so isn't easily to run on remote networks.

Given that we are not trying to do anything more complicated than force a device to send us a packet , and perhaps read a few fields out of that packet , I decided it couldn't be to difficult to roll my own.

So I have the great honour to I present to you, ngdiscover.py - I can't guarantee I've got all the fields right - I've only worked out what they are by inspecting the response packets, but the script works for me.

Have a play and let me know what you find. If it doesn't work for you - you might find the '--debug' option helpful , this makes ngdiscover.py dump the raw response packet to the output as well.

How to store a Freeform games

2010-01-04T20:55:00.001Z

In a previous post , I talked a little about the requirements for writing freeforms games. The post then got a little sidetracked into the requirement for macro parsing without talking much about the context in which that parsing takes place.

In this post I'm going to talk about the storage and object models I used for "To say nothing of the Groom" - the freeform game discussed in my previous post. That post also discusses the requirements which led me to this model.

Let us start by reviewing the requirements. I need to a store a collection of paragraphs, small words and references. These which may need evaluation by a parser before reaching their final form.

A common choice for a storage backend for the summary data is some sort of Relational Db. RDBMSs are good at storing fixed sized or approximately fixed sized elements and references between elements. But in my opinion they do not handle large chunks of text such as paragraphs well. They can store them - but it not their main strength . Given that I am writing what is eventually to be a document for reading by humans they is going to be a lot of paragraphs , and facilitating these paragraphs is the number one requirement.

Although some writers do use RBMS this use seems constrained to storing summary data which can be used early in the development to ensure your have the a workable game at the basic level. A sort of prototype if you will.

Another possibility was XML but that has problems which I discussed in my last post. I need something simple and straightforward while hopefully being resilient. There are a number of XML using tools for freeform games which already exist on the web such as LARPML from aegames. But none of these shield the use from the uglyness of XML, and I feel that the macro language is I talked about in the last post is just that bit more readable than the XML equivalent.

To be able to automatically generate the final documents , and to resolve the macros though I need to have some code that can automatically find the referenced elements that I have stored. And the simplest way of storing chunks of text is in a file. On their own, one to each file.

Actually this does have a few problems.

Efficiency. Or lack thereof. Most modern system are optimized for files larger than 2K,¹ which translates to approximately paragraphs longer than 450 words. Which is about half the size of the paragraphs I ending up writing. But I should probably learn to write longer and better anyway.
There is an insidious technical problem with using a file system as a database, which if the application writer isn't aware of, can under certain circumstances lead to data loss. More glorious technical detail on on this here. And while that discussion talks primarily about linux the OS and filesystem mechanics on Windows will be similiar.

The second of these problems wasn't well understood (by me at least) at the point in 2008 I was working on my game, and to be honest is not a serious concern when I was using a reliable and well written editor on the files directly. The first is of a greater concern as it means the actual amount of disk space consumed will be much greater than the data stored - particularly in the case of the small attributes. However I judged the savings made by using ReST over winword/OXML etc would more than make up for this.

Actually it appears I was wrong on this TSNOTG, takes about 3.6Megabytes of diskspace (du -sh) for 316K of data (uncompressed zip size). Which compares to around 2Megabytes for the full set of word documents used in a recent Peaky game.

I also note that I can store the entire revision history though of my game in 530K zip file, which can also provide a handy mechanism for file interchange between working partners.

But in a world where 200Gb (that is 200*976M)² drives are common place , and even SSDs on small netbooks are multiGb, it doesn't seem that big a deal. We are a far cry from the days I looked at 100Mbyte Hard disk with envy.

The last remaining part of this is how do I organise the element of these paragraphs - or perhaps this should have been first part.

Given that we have a number of object types, Characters,Plots and other depending on the Game such as Ability cards, Contingency Envelopes, and Item cards, to make life simple each of these types of objects was given their own directory.

To store the rest of the data I decided each object should also be a directory, and for most of the attributes where encoded separately individual files, which is as I have already suggested a little wasteful.

The advantage of this technique it allows me full use of all of the standard tools on my system to edit and search these files. During the writing of the game I found it vary easy to write custom search using snippets of shell.

The final advantage of this method of storage gave me plain textfiles which I could easily store in revision control.

Footnotes

Based on the idea of maximising use of 4K blocks, the most common block size of filesystem in common use.
200*1000*1000*1024 / ( 1024*1024 ) Megabytes. Yes ? Now If drive manufacturers used the same units as the rest of us ...

Status Update.

2009-12-25T00:17:00.004Z

Mystery machine finally got its packfile save and load methods over the Yule weekend. My next task is to move all the existing print statements into a python log object so that the debug messages et al can be managed sensibly. Then will look at a bpython integration so that MysteryMachine can at least be experimented with by extreme geeks.

Writing Freeform games

2009-12-02T21:00:00.002Z

The main sink for most of my free time last year (up till November 2008) was a freeform game set at a Victorian wedding. I ran it with a small number of players -about 25% the available spaces and consequences that November.

This was the first fully plotted freeform I tried to write, although it wasn't the only one I did during that period as it overlapped with my first visit to Peaky.

After getting back from Peaky looking at what I had done was an eye opener. I realised how little I had actually done. I also began to find my everything in one file approach was finally reaching the end of it's usefulness.

Now I normally start projects with an everything in one file approach then as I have ideas I can jot them down in that file and can move things around easily. However after a while it will become unwieldy and I expected that.

The question though became what is the structure I need to move on to. I asked around, some people could apparently use databases others just used word docs and the magic of cut and paste to move paragraphs between files.

Now for those of you who aren't familiar with Freeform games , or the writing of them - Steve Hatherley provides a overview of the process here , and his site contains a wealth of other tips on writing in this genre.

There a couple of interesting quirks to this process - notice you don't use the final document or even a version of it to search for plot problems. You have special plot document which is not part of the 'deliverables'¹ of the project.

Another quirk which is not so obvious is that sometimes quite late in the design you need to change elements of the games which are fundamental the the grammar used tin the descriptions of them, the classic example is the gender of a character - you need to change the characters gender and name because you haven't got exactly the right mix of Men & Women signed up and you need to make a few changes. So you swap a character gender and change it's name and that sorts that.

Except you now have to go through every document you written look for any references to this character and change it those appropriately too. Taking care not to miss any.

You don't need to be a genius to see that there are opportunities for automation here. From search and replace to XML based character definitions , too something a whole lot weirder. Which is of course what I did.

First off lets look at what's wrong with XML and the answer is not a lot but I was writing a game not writing Game writing software.

So I was not wanting to write a huge parser and complex programs to deal with finding the element to replace inside a XML doctree replace it, and reseriallise the XML . Also I find that the XML/HTML/SGML type tags do interfere with the layout on the page - and hence the creative flow a little bit.

I've been using resturctured text , or something similar as a format since before ReST was formalised . So I can think easily in that format - which makes it natural for me to keep my documents in that format, particularly since it has good pdf and html publishing tools.

That leaves actually doing the replacement and as far as I could see a the time ReST doesn't have macro support (more on this in a later post). But it seemed simple to used some unusual character to create and escape to a small macro parsing utility.

I restricted the macro language to just simple value substitution and the ternary operator. The only boolean expression which I needed to implement was equality. The most recent version of the parser alone is here at the Mystery Machine repository.

One additional element was required and that was a script which produced character sheets and plot summary documents. But more on those and the storage model in a later post.

Footnotes

Well technically there they may be deliverables by which I mean documents useful to the running of the game, as they can be useful to the gamesmaster. But the primary documents are the character sheets.

Re incarnation

2009-11-14T13:12:00.005Z

I going to try re-incarnation on this blog.

I hope to post here more regularly about the various Open source work and other computer related geekery that I've been up to. Not that this is a particularly sensible time for me to start this as I am away at the end of next week to go here .

I'm not going to try to keep Roleplaying out of this blog infact I suspect a I may make a number of roleplaying posts here - if I manage to keep remember to post updates.

I'm also going to try to post stuff about project which may be closed source or administration closed source boxes which I come across in my day job of sysadmin-for-hire.

Looking at the date on the last post here, it been 3 years since I last posted here. There is no way I can update you on everything I've done since I last posted, because if nothing else I doubt I can remember everything.

However I may post in the coming few week some anecdotes from the period about such things as spectacular cluelessness on the part of Telco tech support (no change there then) and my experience with commercial embedded devices (running linux ) and the vendors thereof - which definitely deserves a post all of its own.

The will also be quite a lot python here in the last year or so - I have found myself using it quite a bit - more so since lots of the projects I have been working both at work and for myself seem have dependencies written in this language.

I'll leave you with a few teaser link for some of the discussion points which might be coming up. Come look at the lovely sources

This will also clue you into why gaming might be a bigger part of this blog than it has been in the past.

It seems a mad idea.

2006-10-22T16:51:00.000+01:00

I've been trying to write a md personality , in fact I had hoped tho be coding it but I've it's a Satuday Afternoon - and I had lazy morning.

The Sky Plus box, was runing out of disk space agains so there were some important telly watching to do to ensure it didn't fill up - or so SWMBO told me.

Back to the new md personality - zerobad - the aim of this personality is to mask bad sectors and other errors from the higher layers . In general this is not a good thing as it prevents any code in the higher layer from doing its own error recovery - and for journalling filesystems would break the guarantees the journalling subsytems are trying to provide.

So why do it then?

The rationale for this device as come from the amount of time I have been spending doing block copies of harddrives with bad sectors on them. eg.

dd if=/dev/src of=out conv=noerror,sync

The reason for doing this copy is almost always fikesystem recovery of failling disks, and by doing this first we get two really useful features.

We protect ourselves from the disk failling any further during the recovery process as we have a stable copy of the filing system.
We have a mechanism with which we can undo our work by making another copy.
We can always give the disk back to our customer in the same state as it arrived in , excepting any further failures (see 1).
Some recovery tools don't work well on disks with bad sectors in some critical places. This seems to be particularly true for chkdsk.exe - and worse sometimes the windows can refuse to recognise these volumes.

However this approach has a real downside , it takes a substantial amount of time to make this copy and this slows down how fast we can get a users data back for them. The time to first file recovered is a critical one particulary if the user can specify which files are most important.

So as an alternative if would clearly be good if we could mount the original disk in a way which did the same translation as dd's conv=noerror,sync. At that is exactly what zerobad is intented to do.

Specifically zerobad will only mask read and will still propagate write errors so that as far as possible journalling and similiar filesystem guarantee's can be met.

It's worth looking at which of the advantages we originally noted our copy approach has can be still supportted by our new approach.

Clearly isn't possible as we are now accessing the disk in a random access manner, but more on this later.
It seems at first sight we might lose this advantage as well, but in fact the linux kernel already provides precisely what we need to keep this feature - writeable snapshots. If we always use a writeable snapshot to make any recovery changes to we can keep our undoablitiy - in fact we can possible do better than before if we have multiple snapshots.
Given the above we certianly have this ,although out access pattern has chnaged which might affect the chances of failure. I'm not sure there is any way to judge this easily - in simple terms we expect more long seeks, but less actual disk reads than before. I don't have any figures on how this will affect disk faillures.
As long as the disk recovery tools can operate on our snapshot / md personality this is given too.

In fact you can also see we can achieve (1) by doing a backgrund copy of the drive to a safe store , although in this case we stil need enough space to store the entire disk image we are working on which does remove the advantgae the beign able to use snapshots would otherwise give of only needing enough disk space required to store the changes. Another use for this personality has been suggested recently - raid5 recovery as the raid5 system won't automatically recovery from when a disk has failled and a second disk has bad sectors - in fact it could fail the second disk making recovery and even mounting of the array impossible. With the zerobad personality you could mask the bad sector allowing recovery and mounting to continue.

You would have corruptted data in the bad sectors of the array but at least you will have the majority of the information intact.

This use is not for the faint-hearted but any sort of diskrecovery is problematic and I tools like zerobad while being dangerous in the hands the inexperinced can be godsend at other times.

And this is for unix if we wanted to be handheld we'ed probably not be running linux in the first place.

Finally the eagle eyed amongst you will have noted I mentioned a recovery tool a different operating system which I would like to run on the new volume. There are a number of approaches such as virtualisations and iSCSI which make this possible but the discussion of this is outside if the scope of what I wantted to talk about here.

This week I've been mostly thinking about cryptograhpy..

2006-04-20T21:04:00.000+01:00

SSL

At work I've been trying to make an VNC work using SSL connections. Fog Creek did it it their copilot software but the it looks quite hard to disentangle this from the other changes they have made. And it is useful when selling the idea of remote control to be able to tell people that they've got the same security levels as their internet banking uses. Or so marketting tells me.

Unfortunately while VNC works really well even over low bandwidth links without stuttering, when I use it with SSL I'm getting data stalls.

It is quite subtle because it the current implementation I have to use a special forwarding agent at our firewall. As a result I've been starring at the internals of openssl, packet traces and even schneier so I can spot any obvious goofs. (And yes I am stting TCP_NODELAY - which improves the responisveness but doesn't prevent stalls.).

Mental CCG

I said I got Schneier's AC out, while techincally true I hardly read it for the above although I did dip into to during the journey to work. I stumpled across the page on Mental Poker. Which given my current interest in CCGs, well l5r specifically - I started to think on whether the protocol could be used to play l5r.

My original conclusion was basically yes. In fact you don't even need to use a public key algorthim with the property:-

             Da(Eb(Ea(M)))=Eb(M)

  Da(c) = Decryption of c using key a
  Db(c) = Decryption of c using key b
  Ea(c) = Encryption of c using key a
  Eb(c) = ENcryption of c using key b

which mental poker needs.

The reason for this is really simple, in most CCGs every body draws form there own deck. (Well in l5r decks as each player has two decks of at least n(=40 usually) cards).

Hence you end up with a protocol like this. I'm only going to show one half but you need to do the same again with Bob and Alice reversed for a two player game. You could extend it to n players but that starts to allow cheating by collusion.

Alice creates a new private/public key pair for the game.
Alice publishes the Public key, which she uses as an encryption key a.
Alice creates one messge of the format Ea([CardId.RandomNo]) for each card in her deck. CardId here represents the name and other details of the card.
Alice send this list of messages in a random order to the player on her left (Bob).
Everytime Alice draws a card Bob choses a card at random, and tells Alice which one he picked. If this card is going into Alice's hand Bob rememebers Ea([CardId.RandomNo]) is in alice's hand.
When Alice reveals a card from her hand She tells Bob Ea([CardId.RandomNo])=[CardId.RandomNo].
Bob verifies that Ea([CardId.RandomNo]) should be in Alice hand (has he has seen everything going in and out of the hand in it's encrypted form. And can use Alices public key to also verify Ea([CardId.RandomNo])=[CardId.RandomNo]. This ensures Alice can only reveal cards from here hand the which Bob has chosen randomly out of here deck.
Other cards can be revealled in a similiar way, although in the case of card which remain face down for all players Bob cannot reveal Ea([CardId.RandomNo]) else Alice would know what was on the face of the card. The is important for card facedown in provinces for example.
If cards need to be reshuffled, all card which are being shuffled together are indicated to Alice (This card be shuffling a card in play back into the deck, or even facedown ninja) and she creates a new randomNo for each and passes back Ea([CardId.NewRandomNo]) for each card.
Bob should probably remember the old and new sets of Ea([CardId.Nonce]) for verification later when Alice reveals here public key.
At the end of the game either of the following needs to happen.
- Alice reveals here private key and Bob verifies that all the assertions about sets of cards he has remember and the deck of cards is a legal deck. Bob must also do the same.
- Alice and Bob collects their assertions including the orignal encrypted deck list, and his private key and information about who one. All of this in then sent signed and encrypted to the TO who verifies the assertions and that Alice hasn't changed decks between rounds.

Actually it would be possible for Bob to verfies then deck hasn't changed without the TOs help if the TO published a signed hash of the decklist. Bob can reconstruct the decklist if Alice reveals her private key. However if you not the TO only use is to help collect the results and verify a against cheating so Alice can do so without revealling the contents of her deck. Since it is obviously impossible without new some really clever crypto to verify a decks legality without seing it if Alice doesn't want to reveal here deck a TO is needed.

In my play experience I suspect most casual games players will be happy to reveal ther decks or take cheating on trust, in tournament there will prolly be a TO who is a trusted third party anyway.

Oh you might have noticed I orginally said the keys didn't need that special property. I was wrong because in l5r at least there is a action which allows a player to look at a certain number of facedown cards. These could be cards on the top of his opponents deck. Unfortunately his opponent doesn;' get to see this cards. So the Poker hand exchange protocol needs to be used exactly as normal here. Which means we end up having the quaratic residue leak problem documented here.

Open ID

I've sort of started as I said before playing with LJ as well as this blog. However it would be nice to offer my readers (assuming I have any) a consolidated feed. Unfortuanely since some parts of my LJ journal require you to be an authenicated person in f world that currently has some difficulties.

However I noticed that LJ talks about openId on it login page , that and the fact I used it and my LJ login to look a someone deadjournal journal page prodded me into digging around and finding some more out.

The OpenId specification turns out to be pretty open and the team is primarly concern in just getting it working which is good.

So given I know I can both this and my LJ blogs as RSS feeds, I might create another blog site but which just gets the approriate authorised feed for the approtiate LJ and this site and presents them as a single blog. The bad news is this is essentially a MIM authentication attack so the site might have to be primarily AJAX or worse a damn program you have to run on you on computer.

Some design is clearly required here. Of course I could move all my content off both blogs onto my own openId compliant blog. But I'm not sure whether my friends who are LJers could add me a friend with just a non-LJ openId -although I believe this will be the case in time.

Finding contentment....

2006-04-17T20:08:00.000+01:00

It's odd I haven't really found much to post about in the last few weeks.

I really think I ought to update this more often. In the mean time I've got myself an LJ account so I can read friends locked posts. Apparently on LJ you can keep posts private for friends. I can understand why though.

Anyway I've hand a contented Easter, spending time with my SWMBO. Much domestic stuff has been done.

The domesic stuff left me in an odd nimd state, so searching through old project I pulled up an old piece of code which I wrote for DOS 'back in the day' as they say. Unfortunately the way it was structed push the limits of he compiler (TP7) I using back then.

About 4 years ago I tried to rewrite the code using 'STL-style' in C++ they I could continue trying to play wiht the new idea the code encapsulates.

Yes, I'm not telling you what it does.

When I first wrote the code I had problems with 'value<->instance' equavalence as I had used STL container to try to store instances. Which of course isn't really what the STL does , as the STL has value containers.

In the meantime I've gather quite a bit of experience and understanding of the STL . So I thought I'd tidy it up and see If I could get it working.

As I said this was an historic project so I discovered that the basic Makefile I had used didn't use -Wall , as a result I missed the function where I forgot to include 'return foo;' which TP didn't need. These days I tend to adapt Makefiles from my last project.

Of course it still doesn't work for some reason the following code doesn't work.

               foo::iterator   end=a.end();
               foo::iterator   point=a.begin();
               .
               .
               
               .
               .
               if (point == end ) { ... }

Why I'm not sure. I need to check the STL spec, or look for a bug elsewhere.

Time passes...

2006-03-01T18:56:00.000Z

I hadn't realised it had been so long since I posted, almost a month. Gosh.

I've spend quite a bit of time investigating the mess that is worldwide timezones. Particlurly to sort out some of the test failures I found when I went back to fix one of the wxWidgets bug I found furing bug day , which I blogged about earlier. There is nothing particularly blog worthy , I suppose in this except why do all the unices seem to use a nice/etc/localtime file and none of them to provide a way for application to parse it.

I'd like to parse it myself becuase wxWidgets wxDateTime class is 64 bit even on 32bit time_t platformss. Sigh. I suppose I could use zdump search trick to find all the discontiniuities but it seems a bit of hack, and difficult to extend DST rules.

Windows of course does something entirely different. In some ways better, in that you can query the API for any date in our valid range, but it uses entirely different timezone names, and it DST rules are much more simplistic.

We going to move over to using reqeust-tracker at work, unfortunately this is going to a protracted change as RT doesn't have any client or customer managment features, and we are going to want to invoice direct from the RT data. This looks like it going to become a 'project'.

Finally we've been asking by one of our customers say $COMPANYA for the sake of clairty - to set them up email for $COMPANYB which is no externally visible connection to $COMPANYA - but they all want it to appear in togother in outlook. We cant hide the fact that $COMPANYA and $COMPANYB share a mail server but lets hope no-one notices this , we haven't been asked directly about this either. But they would like top reply as from $COMPANYB 's domain to emails to $COMPANYB - outlook can't actually do this if you are using it with exchange server. Apparently by design. THere are people who sell highly priced plugins which are supposed to fix this, but I downloaded the trail edition an tested on my test system anc couldn't make it work. Which probably means more custom outlook development . My favourite.

And I'm not mentioning the fact that if you put a $COMPANYB email address in to the rules wizard, becuase it is associated with $USER it changes to referencee in the rule to $USER and then most wierldly of all goes on to match only that users primary address.

More wierd little bugs...

2006-02-02T20:53:00.000Z

A collection of little bugs I've hit over the last few days, and one which I first saw a week ago , but re-appeared today it a different form.

The Outlook is corrupted

Nobody reading this is likely to be surprised that I consider MS Outlook to be seriously badly designed. It manages to give a reasonable 'user experience', but it has appalling consistency and logic about the way it handles it data, not really managing to keep the concept of locally cached copies and masters copy separate. I'll might blog about this some other time - if the rant comes on.

Aside from that we reach todays wonderous behaviour. About a week ago one of my customers rang and said they has some trouble running searches on contacts in the company wide public contacts folder. Checking around it was clear this folder was the only one affected and it affected every machine and every users (separately) . Permissions were changed and things checked , but no resolution was found.

Today the boss at that site called, and complained his Blackberry wasn't sync-ing reliably. It would start the sync reliably but then give up half way through. Luckily however the Blackberry sync has a debug mode, which will show the last item the sync process considers.

This was used and a contact was giving lots of errors when accessed was discovered. A new contact was created with the same data in in it and the old one deleted.

Suddenly both the sync and the data worked fine. Why didn't outlook complain earlier though.

Samba and Fandango on Sid

Moving on , I was installing a new Samba server for a client into its own standalone domain. A pretty standard config for us. As a result I was basing this smb.conf on on I had done earlier. Mistakenly though I allowed a windows machine to connect to the server before I realised I hand the wrong WORKGROUP/DOMAIN name in the config.

But I could just change it in smb.conf without a problem.

Couldn't I ?

For some reason as I began to notice as I was doing the rest of the install the old name kept popping up in odd places on the windows machines.

Digging around I bit I discovered that the samba would still give me a SID for that name if the 'net' command was used. Even worse windows cached credentials weren't working which mean the laptop could hardly be used away from the office - which defeats the point of having a laptop in the first place.

Something had to be done, there was only one thing which seem to hold any promise. Empty the DOMAIN of members - delete samba's sid database. (Actually I killed all of /var/lib/samba to be sure) and re-imported the use profiles. Phew

But the real question is why didn't samba just regenerate a sid for the new domain name , when it could not find one in the database? OTOH, I did have to regenerate the user profiles as the permissions part in HKCU\ was rather screwed on the old profiles before I did this. This points to the possibility of interesting behaviour if you regenerate the SID too easily. But you know, I still think regenerating the SID is probably safer though. Microsoft warn you never to rename stuff - you can see why they just ban it if you look at those issues.

An Enlightening experience

We are moving over to dual-head Xinerama setups for our office PC as we now an awful lot of work via rdesktop and VNC, it make sense to have a monitor for the customers desk and another for your own websearchs and may be a working vmware install of the same OS as a comparison.

Anyway some nice radeon 9550 video cards were purchased which were quite cheap and promised to support dualhead on the xorg drivers at the very least.

So on Friday I managed to install the card, set it up and get another monitor unpacked an on my desk. I even worked out the dualhead xorg.conf file.

Then I logged in an while enlightenment my windows manager was starting up the machine locked solid. .Great. Anyway further debugging showed if I ran metacity instead I could log in fine.

Unfortunately I'm quite attached to enlightenment , I really quite like it brushed-metal theme, particularly the way the options I most commonly use are on the titlebar, and the widgets on the titlebar can be expanded to see the others when needed. The fact I user a virtual desktop with multiple workspaces is another feature which new WM seem to have lost.

sigh. One day I'll get round to my rant and UI designers - who remove the useful features be case they are to confusing. Synaptic for example as lost a nice simple technique which allowed easy selection of which distribution you want to install form - on a package by package basis

I've got a ltrace of what enlightenment was doing immediately pre-crash so I'll wander over to debian-x and see what I can find out.

MJ Ray: Tour de France 2007 in London

2006-01-29T19:46:00.000Z

Mj ray says: MJ Ray: Tour de France 2007 in London: "TdF Blog mentioned the announcement that the 2007 Grand Depart will be in London, England (official announcement at Red Ken's site, official London site) . Of course this isn't this first time that the Tour de France has come through England, it doesn't seem that long go it came through Crowborough , my home town. Of course watching it go by with a pint in hand wasn't to bad for the 3 seconds, the leaders pack was actually in view.

It's the little things which make a difference.

2006-01-24T21:43:00.000Z

Like forgetting to check to permissions on "/".

Yeah, well I was putting together a new computer and since I had built the required configuration before I collected all the approriate files on my workstation. I then tar-ed up a fake /etc and /usr/local directories in to a single tar file and took it over to the new machine, Cd-d to root and unpacked it on.

Of course tar helpfull chmod'd the new system's "/" to the fake "/" I was using when I put the tar file together. But I didn't notice this at first.

The thing is nothing stopped working straight away. It was only when I rebooted and exim failed to start with the meaningful message "user mail was not found" that the problems started - I tried ripping exim4 out of the system entirely and putting all back in , fighting the debian packaging system as the would normally leave loads of broken packages.

I ask one of my colleages to look at it and he point out that he couldn't su to mail. I had spotted this but was sure hoe to debug it - you often cant strace su for security reasons.

While su's error message as lots more help, "Can't cd to /var/mail' I had checked the /var and the /var/amil perms.

Suddenly as I was explaining all this to my colleague , the penny dropped and I check "/"s perms and all was clear.

A Day for Bug

As an occasional, and the past more frequent contributor to wxWidgets . I decided to help out when I saw the Bug Day on Saturday annouced. Even though it was me & my girlfriends anniversary I managed to spend a few hours helping out.

More interestingly it has awakened my enthuasim again - so maybe I'll spend a few more weekends just picking of bugs in the bug tracker and doing something about them.

So the even the bug day didn't no much on it own , it may have got me and hopefully some others moving again , or faster which will help overall

GPGkeys on usb

2006-01-08T14:06:00.000Z

I found this blog post (which is unfortunately down atm) Matt Brown: New GPG Subkey / Offline for a few days

One of my Christmas presents was a brand spanking new USB key. For quite a while I’d been planning to put my SSH / GPG keys off onto something like this for some extra security.
I’ll write up the exact steps of how I achieved my setup in the sometime next week, but it was all fairly easy.
[snip]
All in all I’m quite liking the new setup, plug usb key in, password dialog pops up for SSH key, enter password, password dialog pops up for GPG signing key, enter password. I then h"

I'm quite interested in seeing what Matt has actually done later, but I've been considering using one of these to actually put the agent code into them.

The private key never leaves the usb device at all - although it is obviously must be saved in some portion of the device. But I would make the key write-only. I would probably also put a physical button on the device which has to pushed for the agent to run. This is to mitigate the weakness in agent-forwarding schemes where root on any machine in a forwading chain can communicate to the real agent or agent device. (This means a root user can masquearde as you. It works because root can always get r/w access to the agents commications channel). By forcing the button to pushed for each authenication attempt it makes more likely that attacks are at least visible to the user.

Unfortunately I understand there is a 'bug' the ssh-agent protocol and it can leak the private key. Which limits this.

On Other things

2006-01-05T20:57:00.000Z

One of the small pleasures my job gives me is that while travelling to customers sites (by car), I often find myself listening to radio programmes I might otherwise I have missed - or indeed not thought to listen to.

Today I happened across one of those rare gems in the the radio 4 schedule. Entitled 'The World's First Rock Band' it was a wonderful look at - well not really rock music, more what pterry might call "music with rocks in ". Specifically music played on instruments made at least partially from rock - the geological kind in case of any remaining confusions.

Follow the link while it works as the programme seems to be available for listen again.

Don't underestimate the resourcefulness of people being annoying

2006-01-05T20:52:00.000Z

Mostly my day jobs involves sorting out unusual computer problems, most often they invovle the Windows OS. I can across this - Don't underestimate the resourcefulness of people trying to be annoying.

Anyway the author talks about removing configuration API, and documentation of where those settings are stored in the registry. This is so that they cant be abused by installers. The idea is there is that an installer shouldn't change a users settings. This is all for the good and all installers should follow such guidelines. Mind you I'm not aware of such a guidelines document from microsoft , which specify what things to leave alone in your installers.

But by not documenting some of these things - how is a local system administrator going to fix them when the something goes horribly wrong. A simple example would be carefully constructed registry entry which causes the document UI program to GPF when it reads/processes the settings, but the main library,which is the user of the settings still interprets the settings in the way the mal-installer wished.

A local sysadmin is going to have merry hell trying to sort such a situation out, because he can't even look up on msdn or technet where the compromised settings are likely to be.

The bug system

2006-01-03T21:06:00.000Z

Well, a grand new year was had, I went over to a party some friends were holding to discover they all seemed to know each other purely by reading each other LJ. Anyway I said I'd post about the bug system I was sorking on some here or some docs culled from the development directories. It pretty much works as described although I have yet to actually use it anger - and it still need some debugging itself. I haven't even tried the search yet, although I have coded an algorithm which should work as per described . (Famous last , I know). PS: - I can believe how long it took me to compose this , especially since I started with well formed html, produced by pod2html. Sigh. And the html seems to have got worse not better..

Semantics for the Bugbase data

Record Format
Branches

Manual Page for API

SYNOPSIS
DESCRIPTION
METHODS

Semantics for the Bugbase data

Record Format

The 'current' status of a bug is tracked using a record object, which is stored in a file. This file is checked into the SCM/VCS system alongside the code.

The dates the bug was first committed and subsequently revised reflect the date the information in the update were made available in the bug database. So it important not to use commit dates for bug records in to much of a similar way to you might for code files.

If you consider the usual lifetime of a bug this make sense - a bug is coded at some arbitrary point in the development cycle but only some time later is it discovered and added to the bug database. When a bug is fixed we leave the bug record in the VCS and change it status 'Fixed'. And similarly for other terminal states.

By using symbolic tags to indicate which versions of the code suffer from the bug, this means we can only retrospectively track bugs against tagged versions of the software. But this is not unreasonable as most , if not all development shop use tags to mark 'versions' of the software , which are used outside of the development team itself.

This gives us or first SCM requirement - our SCM system must provide us with a mechanism for adding a existing tag to new files, some distributed SCM systems may have a problem with this concept. Indeed in some people's view this is a violation of proper SCM semantics - as you should be always able to track all changes to the data /and/ metadata a SCM system stores. In pure software terms adding a file to an already released version is clearly a stupid thing to do as it changes to exact build used by that version.

However these semantics allow the development team to track bugs by date in the current version and by tag in earlier versions. For SCM system which don't support this , we should be able to simulate it with extra tracked data, which contains the bug revision to symbolic tag mapping. (TODO).

A desired side-effect of this semantic is that new versions (tags) of the software will automatically be assigned with the current status of all the reported bugs, so if the bug has been fixed the new version of the code will still show the code was fixed, and similarly if it is still open the database show the current version as having that bug.

Branches

Unfortunately the above would suggest there is a problem if someone reports a bug in 1.x which has already been (maybe unwittingly) fixed in (say) 2.0 . At first site this seems to be a problem, as it would show 2.0 has having the bug - assuming you want to keep the bug open for fixing in 1.0.

But as you can see in this case 2.0 would have to be on a separate VCS branch to 1.0 otherwise you wouldn't be able to continue to track changes to 1.x .So in this case you only need to create the bug record in the 1.x branch.

However, the Corollary to this that if the bug exists in both versions you need to create separate bug records in both branches. Of course this happened automatically for bug which exist when you branch, as your VCS system takes care of this.

OPTION:1: The system currently provides no tools to keep these records synchronised.

OPTION:2: To keep both bugs synchronised you need to decide on of the bugs is the master and create a reference record on the other branch. : Problem , this is not what standard branching does, so if bugs which are creating before a branch point end up as back to option 1.

Bugbase API

Bugbase::Bug - Operate on Bugbase bugs

SYNOPSIS

    use Bugbase::Bug ;

  my $bug = new Bugbase::bug(Summary => "bug title",
                             Severity => "Baad",
                             Status => "Open");

  $bug->save();
  $bug->version($version);

  my $otherbug = Bugbase::bug::find($pathlist , @findspec);
  print $other->version();

DESCRIPTION

Provides methods to manipulate the bug records with bug management centric names.

This module is basically a wrapper around either VCS::xxx , and DB::rfc822 which does the approriate thing.

This removes the need to thing about the bugs as anyhing else and just thing about the operations in terms of the bugs, rather than how this might be expressed at the back end.

METHODS

new: Creates and returns and new bug object, for a new bug.
find($pathlist , @searchspec ): return a bug object matching the arguments .; Path list is an arrayref of paths to search, and search spec is an a array of hashrefs. Each hashref contains a key/value pair where the key is the field to be matched - the value is a an array of regexps, to match.; The specification is considered matched if all the regexp's in any hash set match.
Status: An accessor for the status method of the bug.; This accessor clears the version array, so that $bug->status('Fixed'); doesn't mark the bug as fixed historically.
Resolution Priority
Hardware
Os
Creator
Assignee
Summary
Commentary: Accessors for the simple bug fields, these are store in the bug file. The only exception is commentary which is stored in the bug record's body. You can use any fieldname here which is compatible with DB::rfc822.
version: This is an accessor for version array which constains a list of 'tags' the currently selected bug revision. For instance if the currently selected revision has status 'Open' , this will return an arrary contain the versions of the software which are believed to contain the bug.; FIXME
1.This method cannot currently be used to clear the tags on a version. 2.This method currently operates on the HEAD.
revision: This returns the revision number of the selected bug record.
commit ( message ): Save and commit to the repository the currnet chnages in the bug record.
fullname: this returns the full path name of the bug file.
shortname: this returns the leaf name of the bug file.
canonicalname: this returns a globlay unique a staticname for the bug, it is normally based in the VCS naming conventions.

Hashes and probability

2005-12-26T10:45:00.000Z

Christmas Cheer

Due to my family being made up generally of quiet people, we sort of all desserted the sitting rooom after the xmas meal , and went and did our own thing, well we pop back seperately at times to watch different stuff on the tv and keep Gran company. But I think most of us weere just looking for some peasce to read or whatever , and while there was some good stuff on TV there wasn't much of it. And since the TV was on for Gran - well some of us left her to it. I ending up in the home office reading up on SCM systems, as I'm got a idea presented later about a bug tracker and wanted to be sure it would work on most SCM. In hindsight I think it has to, but I'm being cautious.

Hashes and Probability

Anyway as a result I was reading the monotone documentation, Monotone is another DSCM system and looks quite cool, although shares a lot with git , as far as I can tell, but is probably slightly easier to learn. Anyway monotone has a section in it manual which tries (and to some extent succeds) to defend 'compare-by-hash' . Now for those of you who don't know compare by hash is the technique of usings Cryptographic hashes to todo comparisions, Rsync for one uses this to good effect , but both git and monotone use it to name file revisions. So if two files every have the same hash the system breaks down because it can't tell them apart.

The question is how likely is this ? And it obviously depends the amount of data stored - specifically the number of hashed objects. While the defenders are correct that it not very likely I'm a little woried they seem to useing the 50% threshold in there quotes, eg, the number of hashes required for a there to be a 50% chance of collision. I think I'd prefer it it we culd standards on say the 5% and 1% level similiar to other statistcially uses. Having said that bc suggests for SHA-1 thats still not as bad as it sounds:- (Approximation formula from wikipedia - I couldn't find it on mathworld )

sqrt(2*(2^160)*l(1/(1-0.05))) 387208558096282571848629.44632292079366906761

Which I make to be about 3 * 10^42 objects, and only about half that for the 1% level.

So I'm reasonably happy still with this technique

I'll blog later on SCMs and bugs and maybe more on this too.

First Post

2005-12-24T22:33:00.000Z

Well this seems to be here and working. Testing ,testing 1 2 3. It seemd a little easier to setp than bloxsom... Lets see what happens.