The windows API

I'm not in general a fan the win32 api, which for those of you who know me well won't come as much of a surprise. But while working on a the bottom layer of the ACID complaint store stuff for MysteryMachine , I realised that I could just use TxF on versions of windows that are late enough. And indeed it seems stupid not to, so I decided to rewrite just the bottom layer of the transaction handler to use TxF when available.

Now for those who don't know , MysteryMachine is written in Python which is not the most natural language for direct access to the Win32 API which is primarily called from C/C++. There are some bindings also for C#, but I get the impression there is quite a lot between C# and the kernel when used this way.

So it became time for me to see how I could call this functions easily from Python. I looked into the Python toolbox and found ctypes which is perfect for the job. You can literally just mention a DLL by name, and construct a tuple of arguments. The arguments have to be made from special ctype objects which represent the types primitives available in C. You can read the documentation and the link above to find out more about ctypes itself.

Now I just wanted a simple wrapper around the TxF functions so I didn't have to go through all that rigmorole of type conversion each time, so it seemed that each function I wrote would be of the form of:-

def Windows32Function(*args,**kwargs):

type_list [ HANDLE, LONG , LONG, POINTER, LONG ,LONG, HANDLE, LPCWSTR ]
    out_args = []
    for arg,argtype in zip(args,type_list):
         out_args.append(argtype(arg))
  
    ctypes.dll.Windows32Function(*out_args)

Now the think I really don't like about the Windows Api is the large number of arguments that windows functions regularly take , and which 99% of the time you can just you the same default option, and it struck me - Python has first class support for keyword arguments, that what the kwargs argument above is for.

So I came up with the following:-

def do_args(args,kwargs,name,dflt,factory = None):
    if args:
        dflt = args.pop(0)
    val = kwargs.get(name,dflt)
    if factory:
 val = factory(val)

    return val


def CreateFileTransacted(*args, **kwargs):
    args = list(args)
    lpFilename = do_args(args,kwargs,'filename',None,LPCTSTR)
    dwDesiredAccess = do_args(args,kwargs,'desired_access',const.GENERIC_READ,DWORD)
    dwShareMode = do_args(args,kwargs,'share_mode',0,DWORD)
......

Looking at the code above , I can use the do_args function to take the grunt work out of processing each argument. , You should be able  see that do_args takes an keyword name, a default value and a type in that order, as well as the arguments lists from the original function.
You might be wondering about the first line of the CreateFileTransacted wrapper, it is important that we use a args in a list forms as the tuple we start with (as all tuples in python are) is immutable so we can't pop the arguments from it as we go.

It takes an argument out of the positional list, check the args name in the keyword list, (keyword overrides positional in this implementation) and if neither is there will use the provided default. Then finally it casts it into
the correct type for ctypes to use. Here I name the arguments before I call the DLL entrypoint but thats not necessary a list can be built up like the original version did.

What's really good about this is that do_args handle all of the marshalling ctype requires the use to do, so is quick to write and the code is simple enough to be easy to verify. But the really amazing thing about this is it makes the windows API much more pleasant to work with. I managed to get most of the TxF implementation included all the windows functions done in mainly a day. There has been lots of messing around since to check it multiple platforms and deal with the inconsistencies but it was much quick to code windows calls this way. There are two big contributors to this ease which I don't normally get with windows programming and the first was the simple interactive nature of Python , I could very simply load the wrapper modules into a python console session and start calling the  Windows API directly.

Normally this would be a pain in the proverbial , as the number of arguments is a large and would each have to be specified in the right order and with the right number of intervening NULL arguments for the features unused, but because I could really on the wrapper providing me with all the sensible defaults I only needed to provide the one or two arguments I was actually using it became much quicker to find out exactly what the real behaviour of windows was. I actually found it felt like windows behaved for once - I was quite shocked.

However as I was to discover while TxF behaved fine, windows file semantics meant that I had more problems head, but I save that for a later post.

New toy...

I have just received a Kindle, bought somewhat cheaply from ebay, and I'm looking forward to modifying the running software on the device to add an Ssh shell, and remove some of the reporting to Amazon features.

The simplest way of stopping (a non-3G one like mine) from reporting to Amazon is to non-connect it to a wireless network. But I'd quite like to get a terminal running on it and a ssh client so that I can use it as a display for some other tasks when sitting outside. Unfortunately this is a little pointless if I don't connect to a wireless network.

In the bright sunlight LCDs often don't have enough contrast to compete with the uncontrolled nuclear reaction in the sky - eInk being a reflective as opposed to backlit-transmissive display technology should cope much better. I'm a little concerned the display's speed won't really be up to this use , and that the keyboard will be fiddly - but I am willing to try . It the best way to actually find how usable it is after all.

In the mean time I've copied the novels and papers I already have in pdf and mobi format onto the device (via USB) and have spend some time using it as a simple reader. One problem with the display I have noticed is that due to the electrostatic nature of the display it tend so attract dust and hairs, (particularly cat ones) on the display surface. The surface is so immaculate-matt white normall these are much more distracting than on a LCD/CRT . The other reason for this is while it is easier on the eye as the image forms on the surface of the display (CRT/LCD form the image just inside the display) so any dirt that falls on the display is in focus as you are reading the kindle.

On a side note about those looking for MysteryMachine updates, I have written a transaction manager for a 2PL locking scheme , which does the hard work for the ACI of ACID compliance. For the 'D' (Durability) I need to augment the store API as the store needs simple atomic write/undo/checkpoint type features. This is easy to implement on a store backend which already supports transactions , but harder otherwise. I'm also worried about the performance impact this might have - the store is already the slowest part of the system.

Until I've worked out the store API there is little point in commiting what I've got since it isn't invoked by anything yet. The Transaction manager is a per-system object so in the future alternate transactions managers might be possible (a 2V/TWR oneloks like promising idea.)

MysteryMachine Update...

2Version 0.14 is tagged in mercurial, I need to create the distribution archives and upload it.

This was going to happen this weekend, but I left my laptop which is my primary development machine for this project in thne office on Friday. I've been working on my partner's 'spare' laptop an EeePc 901. Which after some heroic work with aptitude will run the required software - it's not fastest , or most comfortable machine to use - if nothing else just due to it's small form factor.

I'll update the trac site with a copy of the zip and tgz soon, meaning the early part of next week if all goes well.

I still haven't decided on what to do about project hosting. Ho Hmm.

I have also discovered a serious bug involving inheritance and list attributes, there is a unit test for it but that has been passing becasue of a bug in dict_store (fixed in my working copy), this looks like its going to be an awkward one and might trigger the complete rewrite of the Attribute update path and store API which ahas been in the back of my mind for a while. If it does look forward to the something like ACID compliance at that level. I will probably add some sort of transaction support too.

This for the average user this means we can guarantee better predictability of behaviour when you ask the system to do things it can't do, it wont corrupt your data. And should improve, make simple undo paths easier to code.

It strikes me I haven't posted about the state of MysteryMachine for a while, in fact well over a year.

Well during that year good things have been happening , which if you have been paying attention to the repository history you might have seen.

First off all during the months of September and October I ported the Siege Mentality game written by J. Tuomas Harviainen for use as an example game for mysterymachine into the MysteryMachine database. I then ran the game at Consequences using only MysteryMachine to produce player character sheets.

Some more work is need to streamline this operation but actually using the program in anger at this stage was a very good thing to do, so I could see the what are the most important missing features are.

Indeed I wouldn't have been able to even do that without one import last minute feature, The Bidirection reference. This important feature is a special new attribute type which allows two mysterymachine object to be linked together - to make this work a bidilink - as it is known, uses another new feature - the two stage commit - to keep a pair of attributes in sync. This attribute is needed because each end of the link is modelling by its own attribute, this means each end of the link has a handle which can use to follow the link in the opposite direction, while allowing the code to ensure if one end of the link moves the other immediately reflects that change. A link which is moved forces the other end to point to the None object to indicate that is is currently unconnected.

Unconnected link attributes such as this are known as anchorpoints, and a link can only be connected to another anchorpoint. This is because an anchorpoint is not just an empty node, but it also stores which of it's ancestors in the database any link which partners with it will point to.

To give an example a character object may have a list of anchorpoints called 'plotroles' , each of these anchorpoints are places which you could connect a link from the plotrole to the character which takes that role in the plot. The anchorpoint in the character object knows that when the link from plotrole is followed that the follower wants to end up on the character, not the anchorpoint , or the list.

The module which implements this currrently has some pretty good examples in it's documentation , and I'll put them on the MysteryMahcine wiki which is also overdue an update.

The best news is there is now a UI. This means it can start to be used by ordinary people rather than programmers. Up till now you have only been able to manipulate MysteryMachine games and system via actually exceuting python code, but now there is a UI which should work on Mac,Linux & Windows - although I have currrently only tested on linux.

There is a plan for myself and my partner to write a full game in in time for November but we shall have to see how that goes.

Trac or bitbucket.

Updated: 12:55 : I was attempting to blog from my andriod phone , but it posted it live rather than to drafts.
I've noticed that a recent server upgrade has broken the Trac install used by mysterymachine.
Trac was never very fast or responsive on my server and has no account creation support as standard (it is available as an extension). All accounts needed to be created by the site administrator.
This puts a roadblock in the way of causal users trying to report bugs. Not that I've hand any yet but I want to be able let all comers report bugs.
For sometime I've also had a mysterymachine archive on bitbucket, since bitbucket also provides a wiki and bug tracker as well as a large community. I'm strongly tempted to make that the new primarily for MysteryMachine. If I do so I will migrate the existing wiki content there. I will still host a copy of the MysteryMachine repo at hg.backslashat.org , so it is just the the issue tracker and the wiki which are really moving. Does anyone have any opinions.

Another Milestone

Earlier today I committed changeset 'ce4f0384c0b329ee85b08b40ee4bd6087f18fb00' to the MysteryMachine repository With the addition of this code the system can now actually be used to generate character sheets and other report documents. There still isn't though a UI, an installer or a decent plugin manager and even the database isn't entirely feature complete. The most pressing of the above is the installer so that got to be my next push. I;m probably going to take a short break , I've got some creative writing for a friend coming up - and unfortunately he has a database with all the background information in which needs fixing first too.

Find and discover switches, well the smart ones.

Like many people I have occasionally forgotten small details when under pressure, and even worse sometime forgotten to write them down. Added to the fact I regularly visit or remotely handle to handle sites setup by other people I don't always know the configuration of a site switches.

Most modern switches have a web interface or at least a telnet one - but serial ports are becoming less common - even if there are connected.

Cisco devices (and some Linksys now), will reveal their presence to you via CDP, just sit on the net with your favourite sniffer running and watch for CDP packets. However Netgear switches don't do this. Netgear do offer a downloadable discovery program - but it is for windows only, so isn't easily to run on remote networks.

Given that we are not trying to do anything more complicated than force a device to send us a packet , and perhaps read a few fields out of that packet , I decided it couldn't be to difficult to roll my own.

So I have the great honour to I present to you, ngdiscover.py - I can't guarantee I've got all the fields right - I've only worked out what they are by inspecting the response packets, but the script works for me.

Have a play and let me know what you find. If it doesn't work for you - you might find the '--debug' option helpful , this makes ngdiscover.py dump the raw response packet to the output as well.

How to store a Freeform games

In a previous post , I talked a little about the requirements for writing freeforms games. The post then got a little sidetracked into the requirement for macro parsing without talking much about the context in which that parsing takes place.

In this post I'm going to talk about the storage and object models I used for "To say nothing of the Groom" - the freeform game discussed in my previous post. That post also discusses the requirements which led me to this model.

Let us start by reviewing the requirements. I need to a store a collection of paragraphs, small words and references. These which may need evaluation by a parser before reaching their final form.

A common choice for a storage backend for the summary data is some sort of Relational Db. RDBMSs are good at storing fixed sized or approximately fixed sized elements and references between elements. But in my opinion they do not handle large chunks of text such as paragraphs well. They can store them - but it not their main strength . Given that I am writing what is eventually to be a document for reading by humans they is going to be a lot of paragraphs , and facilitating these paragraphs is the number one requirement.

Although some writers do use RBMS this use seems constrained to storing summary data which can be used early in the development to ensure your have the a workable game at the basic level. A sort of prototype if you will.

Another possibility was XML but that has problems which I discussed in my last post. I need something simple and straightforward while hopefully being resilient. There are a number of XML using tools for freeform games which already exist on the web such as LARPML from aegames. But none of these shield the use from the uglyness of XML, and I feel that the macro language is I talked about in the last post is just that bit more readable than the XML equivalent.

To be able to automatically generate the final documents , and to resolve the macros though I need to have some code that can automatically find the referenced elements that I have stored. And the simplest way of storing chunks of text is in a file. On their own, one to each file.

Actually this does have a few problems.

1. Efficiency. Or lack thereof. Most modern system are optimized for files larger than 2K,¹ which translates to approximately paragraphs longer than 450 words. Which is about half the size of the paragraphs I ending up writing. But I should probably learn to write longer and better anyway.
2. There is an insidious technical problem with using a file system as a database, which if the application writer isn't aware of, can under certain circumstances lead to data loss. More glorious technical detail on on this here. And while that discussion talks primarily about linux the OS and filesystem mechanics on Windows will be similiar.

The second of these problems wasn't well understood (by me at least) at the point in 2008 I was working on my game, and to be honest is not a serious concern when I was using a reliable and well written editor on the files directly. The first is of a greater concern as it means the actual amount of disk space consumed will be much greater than the data stored - particularly in the case of the small attributes. However I judged the savings made by using ReST over winword/OXML etc would more than make up for this.

Actually it appears I was wrong on this TSNOTG, takes about 3.6Megabytes of diskspace (du -sh) for 316K of data (uncompressed zip size). Which compares to around 2Megabytes for the full set of word documents used in a recent Peaky game.

I also note that I can store the entire revision history though of my game in 530K zip file, which can also provide a handy mechanism for file interchange between working partners.

But in a world where 200Gb (that is 200*976M)² drives are common place , and even SSDs on small netbooks are multiGb, it doesn't seem that big a deal. We are a far cry from the days I looked at 100Mbyte Hard disk with envy.

The last remaining part of this is how do I organise the element of these paragraphs - or perhaps this should have been first part.

Given that we have a number of object types, Characters,Plots and other depending on the Game such as Ability cards, Contingency Envelopes, and Item cards, to make life simple each of these types of objects was given their own directory.

To store the rest of the data I decided each object should also be a directory, and for most of the attributes where encoded separately individual files, which is as I have already suggested a little wasteful.

The advantage of this technique it allows me full use of all of the standard tools on my system to edit and search these files. During the writing of the game I found it vary easy to write custom search using snippets of shell.

The final advantage of this method of storage gave me plain textfiles which I could easily store in revision control.

Footnotes

1. Based on the idea of maximising use of 4K blocks, the most common block size of filesystem in common use.
2. 200*1000*1000*1024 / ( 1024*1024 ) Megabytes. Yes ? Now If drive manufacturers used the same units as the rest of us ...