How to store a Freeform games

In a previous post , I talked a little about the requirements for writing freeforms games. The post then got a little sidetracked into the requirement for macro parsing without talking much about the context in which that parsing takes place.

In this post I'm going to talk about the storage and object models I used for "To say nothing of the Groom" - the freeform game discussed in my previous post. That post also discusses the requirements which led me to this model.

Let us start by reviewing the requirements. I need to a store a collection of paragraphs, small words and references. These which may need evaluation by a parser before reaching their final form.

A common choice for a storage backend for the summary data is some sort of Relational Db. RDBMSs are good at storing fixed sized or approximately fixed sized elements and references between elements. But in my opinion they do not handle large chunks of text such as paragraphs well. They can store them - but it not their main strength . Given that I am writing what is eventually to be a document for reading by humans they is going to be a lot of paragraphs , and facilitating these paragraphs is the number one requirement.

Although some writers do use RBMS this use seems constrained to storing summary data which can be used early in the development to ensure your have the a workable game at the basic level. A sort of prototype if you will.

Another possibility was XML but that has problems which I discussed in my last post. I need something simple and straightforward while hopefully being resilient. There are a number of XML using tools for freeform games which already exist on the web such as LARPML from aegames. But none of these shield the use from the uglyness of XML, and I feel that the macro language is I talked about in the last post is just that bit more readable than the XML equivalent.

To be able to automatically generate the final documents , and to resolve the macros though I need to have some code that can automatically find the referenced elements that I have stored. And the simplest way of storing chunks of text is in a file. On their own, one to each file.

Actually this does have a few problems.

1. Efficiency. Or lack thereof. Most modern system are optimized for files larger than 2K,¹ which translates to approximately paragraphs longer than 450 words. Which is about half the size of the paragraphs I ending up writing. But I should probably learn to write longer and better anyway.
2. There is an insidious technical problem with using a file system as a database, which if the application writer isn't aware of, can under certain circumstances lead to data loss. More glorious technical detail on on this here. And while that discussion talks primarily about linux the OS and filesystem mechanics on Windows will be similiar.

The second of these problems wasn't well understood (by me at least) at the point in 2008 I was working on my game, and to be honest is not a serious concern when I was using a reliable and well written editor on the files directly. The first is of a greater concern as it means the actual amount of disk space consumed will be much greater than the data stored - particularly in the case of the small attributes. However I judged the savings made by using ReST over winword/OXML etc would more than make up for this.

Actually it appears I was wrong on this TSNOTG, takes about 3.6Megabytes of diskspace (du -sh) for 316K of data (uncompressed zip size). Which compares to around 2Megabytes for the full set of word documents used in a recent Peaky game.

I also note that I can store the entire revision history though of my game in 530K zip file, which can also provide a handy mechanism for file interchange between working partners.

But in a world where 200Gb (that is 200*976M)² drives are common place , and even SSDs on small netbooks are multiGb, it doesn't seem that big a deal. We are a far cry from the days I looked at 100Mbyte Hard disk with envy.

The last remaining part of this is how do I organise the element of these paragraphs - or perhaps this should have been first part.

Given that we have a number of object types, Characters,Plots and other depending on the Game such as Ability cards, Contingency Envelopes, and Item cards, to make life simple each of these types of objects was given their own directory.

To store the rest of the data I decided each object should also be a directory, and for most of the attributes where encoded separately individual files, which is as I have already suggested a little wasteful.

The advantage of this technique it allows me full use of all of the standard tools on my system to edit and search these files. During the writing of the game I found it vary easy to write custom search using snippets of shell.

The final advantage of this method of storage gave me plain textfiles which I could easily store in revision control.

Footnotes

1. Based on the idea of maximising use of 4K blocks, the most common block size of filesystem in common use.
2. 200*1000*1000*1024 / ( 1024*1024 ) Megabytes. Yes ? Now If drive manufacturers used the same units as the rest of us ...