Musings of a Software Architect

Creating an ebook for the Kindle

2011-06-17T20:44:00.003Z

I have been reading ebooks for years using the MobiPocket application and one ofmy reasons for liking it was the ability to create your own ebooks from sources such as project Gutenberg in addition to buying commercially produced ebooks.

I recently gave in to temptation and bought a Kindle from Amazon. One of the attractions for me was that it also allows you to read ebooks that you have created yourself. At first, I only created simple ebooks but I recently volunteered to help create a Kindle version of the excellent "Architecture of Open Source Applications" http://www.aosabook.org/en/index.html and, in the process, learnt more about creating better quality ebooks. None of this is very difficult if you are comfortable editing HTML and XML so this post is intended to share the techniques that I found useful.

Alternative formats

The Kindle will do its best to display text and PDF files but these do not give a perfect reading experience.

When displaying text files, the Kindle treats each line as separate - it does not run subsequent lines into a paragraph. It will wrap lines to fit the screen based on the current test size so all the text is visible but in some cases the line breaks can be jarring.

PDF files present a different challenge. Because the PDF files are intended to preserve the page layout, the actual text flow is lost. The Kindle has to preserve the page layout for a PDF file and cannot change text size or handle pages wider than the screen. I have a few books in PDF form that I manage to read on the Kindle by switching to landscape format but it is not possible to change the text size. On the plus side, embedded images work fine.

Neither text nor PDF files allow the Kindle to provide a useful table of contents.

In summary, if you have a small text file or a file that is only available as a PDF then you can read it using the Kindle but it will not give an ideal reading experience.

Native format

In order to get the best reading experience with variable text size and a table of contents, it is necessary to generate an ebook in the 'native' .mobi format.

A simple book can be generated using the open source Calibre tool. This provides a full ebook management service but it includes a format converter that can take a suitable HTML file and generate an ebook.

I used Calibre for relatively simple books but I then found that kindlegen gave more control so I switched to that.

Kindlegen is provided by Amazon - Google gave me this link but search for the latest version
http://www.amazon.com/gp/feature.html?ie=UTF8&docId=1000234621 and it is the officially sanctioned tool for generating ebooks for the Kindle. There are versions of kindlegen for Linux, Mac and Windows and the download includes a very useful example. A guide to using kindlegen is separately available by going to Amazon's page on publishing for Kindle http://www.amazon.com/gp/feature.html?ie=UTF8&docId=1000234621 and is well worth getting.

Kindlegen also supports the creation of an ebook from a single HTML file but if you want to create a larger ebook or to include a table of contents then you need to go to the trouble of creating an .opf file and structuring the book accordingly.

Creating an ebook using .opf, .ncx and other files

The best way to create your first ebook based on .opf is to take the relevant files from the kindlegen example and edit those. The .opf file is both a manifest that lists the component files and meta-data and also includes a reading order.

The various meta-data include title, author(s), ISBN number, subject and summary and are all pretty obvious.

The manifest part is a list of files to include. These are mostly HTML files but the cover image (if any) is specified there and the .ncx contents file (covered below). When specifying files to include, simply keep them all in the same directory as the .opf file and refer to them by name. You can probably refer to files in relative or absolute directories if you need to.

One piece of good news (if you are lazy like me) is that you do not need to explicitly include image files in the .opf file. If you include an HTML file that includes an image then kindlegen will pick it up automatically.

The kindlegen guide includes guidelines on cover images - basically go for 600 by 800 in colour and try not to re-scale because that reduces the quality. If you use large embedded images then the kindle will scale them nicely for different screen layout. There is no need to change the image size unless memory usage becomes an issue. Larger than 600 or 800 in either direction will be wasted but re-scaling it yourself may affect quality.

Once you have all of the files in the manifest, you need to set the reading order and the table of contents.

The table of contents is slightly annoying in that you are advised to create it in two ways. You create an HTML table of contents that lists the chapters (or sub-sections) and has links (more below on links). This is to be found when the reader starts at the beginning of the book. The second table of contents is an XML file with a .ncx extension that associates text with a file and optional location. This table of contents is the one found when the reader selects go to table of contents.

As with the .opf file, the best way to create the table of contents files is to edit the examples provided with kindlegen.

Once you have the .opf, .ncx and various HTML files, you can simply run kindlegen to generate a .mobi file. Copy this to your Kindle (use the USB cable, it is your friend when generating ebooks). You can then check the tables of contents and cover picture as well as the actual content.

Links within an ebook work simply. If you have a link with the full http:// protocol then following it will open a browser (unsurprisingly). If you leave off http:// and just refer to a file name or an id then the link is internal to the book.

This allows you to build the table of contents or to put links within the book to other sections or to pictures or tables. A link of the form <a href="otherfile.html"> links to another content file in the book. A link of the form <href="otherfile.html#atag"> links to a tag with id="atag" in the file. If you have a set of content files from the web with links between them then consider using a script to remove the http:// part.

Formatting tips and tricks

Once you have a book with a structure and a table of contents, you can tune the appearance. You should be aware that it is better to have simple formats rather then overly elaborate ones. The Kindle screen is limited in size and being too fussy about layout can backfire.

One technique that I wished I had learnt earlier is that kindlegen / the Kindlegen has reasonable support for CSS. I have not tried using external stylesheets but I was able to put class styles in the file headers and they worked perfectly. When I started using kindlegen, I assumed that it would not be this capable so I wrote scripts to edit the formats - changing class styles was much quicker and
easier to experiment with.

I wanted to make some text smaller. This needs restraint as the Kindle allows the user to change text size so don't override their choice but we found that monospaced text was larger than the default font and looked better when reduced. Both the <font>> tag and the font-size style worked but you have to go for the smallest font size to have any effect. In contrast, it is possible to have several larger font sizes but they quickly take over the screen.

Horizontal rules <hr> tags work well to divide up text and can be full width or partial.

The various <h1>, <h2>, <h3> tags work and should be used for sections and sub-sections.

If you want whitespace between paragraphs (for example, we wanted space before and after section headings) then <p> </p> provides a gap.

Paragraphs are indented by default (this is normal book layout) but can be unindented <p style="text-indent:0"> - this technique is described in Amazon's kindle publishing guidelines.

The Kindle does not handle all Unicode characters. I encountered problems with some vertical bars, some quotation marks and a character to denote a space. In some cases, these are stored as entities in the HTML and can be replaced by similar characters automatically. Other cases were UTF8 and these can also be replaced by a script.

Not all Unicode or entity characters cause problems. I found the best approach was to leave them alone and then manually proof-read the Kindle version, spot invalid characters and then go and replace all occurences. This risks missing some. Once I found offending characters, I replaced them using a script. if I was creating ebooks more often then I would just build and maintain a script to do this because it would be consistent across all content.

One HTML technique that works really badly on the Kindle is tables. Amazon's publishing guidelines spend a lot of time on this and they are right. A narrow table is OK but a table does not behave well if the table is wider than the screen. Any excess is simply lost. When I was converting a book, I had to manually re-layout all of the tables and I wasn't able to script the conversions.

Dynamically generating SVG in an HTML page

2009-08-25T20:02:00.011Z

In this post, I want to briefly list the less obvious aspects of using SVG to draw a game board. These are notes from my initial implementation so there may well be more polished alternatives.

For this work I used a recent version of Firefox and I have not yet investigated other browsers in any depth so your mileage with other browsers may vary.

My work was aimed at dynamically creating a page (hence the use of Javascript) but the same techniques could be used to create a static HTML page including SVG.

This project uses XML, SVG and Javascript as well as simple HTML. I have not provided any links to information on these subjects; I recommend finding a good book on XML and Javascript but a simple web search generated enough information on SVG.

Embedding SVG in HTML

I did not want to just have a stand-alone SVG page but an SVG picture embedded in an HTML page. This is because I wanted to make use of HTML elements and I also found HTML better for loading Javascript than stand-alone SVG.

I found that I needed to create the HTML page as an XML page so the start of the page is as follows:


<?xml version="1.0"?>
<html xmlns="http://www.w3.org/1999/xhtml">
      xmlns:svg="http://www.w3.org/2000/svg" 
      >
<head>

This provides both a default XML namespace for the HTML (actually XHTML but don't worry about that) and a namespace for use by SVG elements. This is important as the browser needs to be able to tell whether any element in the page is meant to be HTML or SVG. We will come back to namespaces later. If you are not familiar with XML namespaces then it would be worthwhile brushing up on them but the examples below include all the code necessary.

When I served this from Django I had to ensure that the HTTP Response had the correct MIME type as follows:

return HttpResponse(t.render(c), mimetype="text/xml")

If you are not familiar with Django then don't worry about this line, it just returns the page but with the MIME type set to text/xml.

Within the HTML body I simply embedded the SVG picture directly. I did not include it in any explicit graphical or other object.


<svg:svg id="board" width="500" height="500">
  <svg:rect x="10" y="10" width="485" height="485" fill="#FF9933">
</svg:svg>

Note that the elements all use the svg namespace with the shorthand reference defined at the head of the file. We need to give the svg element an id so that we can refer to it from Javascript.

I found that I had to set the width and height of the svg object. I had hoped to have it more dynamically sized but that did not work too well. Although I deliberately did not set the units of the svg element, I found that it ended up using pixels as the units. I was hoping to avoid setting the size in pixels in order to make the page work well with different sized screens and windows but I was unable to make it scale well. I could have investigated further but I chose to pick a plausible size instead. I may return to this at a later date.

In the example above, I included a rectangle. I could have just drawn it dynamically but it helped me during development to ensure that the SVG was drawing properly before I started creating more elements.

If you just put these elements in a HTML page then you should get a page (which you can put any normal text, images etc. as standard HTML) with a picture embedded in it. The next section covers adding more drawing elements.

Drawing SVG elements with Javascript

In this section, I assume that you are already familiar with using Javascript to generate HTML elements and dynamically add them to an existing page. This type of DHTML is covered by many articles online but the basic concept is to create a new element object of the required type, set any attributes and then to append it as a child element under a known location. The code below uses the same technique but with an extra wrinkle for SVG. If your Javascript DHTML knowledge is limited to copying inner HTML then this is the next level but it is not too difficult.

For my purposes, I knew that I was going to create a lot of SVG elements so I wrote a function to create an arbitrary SVG element.


function createSVGElement( eltType, properties) { // Create an arbitrary SVG element
    var tgt = document.getElementById("board");
    var elt = document.createElementNS('http://www.w3.org/2000/svg', eltType);
    for ( prop in properties ) {
        elt.setAttribute(prop, properties[prop]);
        }
    tgt.appendChild(elt);
    }

This code has three parts:

1) The createElementNS() function is used to create a new element. If you have worked only in HTML then you may have used the CreateElement() function. This version adds a namespace argument which we need in order to specify an SVG element. Without this, the browser will not recognise the element, even if you create an element with a name that is not valid in HTML.

2) The attributes of the new element need to be set. These attributes will include geometric attributes for location and size as well as fill color etc. I used the array approach simply because of wanting a re-usable function. If I was using inline code to create the elements then I could use SetAttribute() as above or use the dot notation. If you use the dot notation to set attributes then be aware that some SVG attribute names are not legal Javascript names (they include a dash). For these attributes, the SetAttribute() method is necessary as it uses the property name as a string.

3) Finally, the new element is appended as a child to the top-level svg element. This is just the same technique as is used for dynamically adding HTML elements to an HTML page. In my example, I added every element directly under the root svg element but there is nothing to stop you from creating a less flat element structure. This might be particularly useful if you want to try changing properties of grouping elements for dynamic effects.

Given this function (and I can already see at least one way of optimizing it), here is a chunk of code that will draw a simple line.


var lineProperties = {'stroke': 'black'};
lineProperties.x1 = 10;
lineProperties.x2 = 10;
lineProperties.y1 = 10;
lineProperties.y2 = 490;
createSVGElement( 'line', lineProperties);

This chunk is slightly clumsy because it has been taken from a more dynamic function. The lineProperties object is initialized with the line colour and then properties are added for the x and y of each end.

Note that I have used both the string way of setting a property and the dot notation (which is fine for these properties). In this case, I could have just created the properties in one statement.

Given the set of properties, we simply call createSVGElement() with the name of the element and the set of properties.

Full details of the properties of SVG elements can be found in the SVG specification.

Associating events with SVG elements

The code above shows the techniques required to draw an SVG picture within an HTML page. In my case, I wanted to draw a Go board and I was simply able to draw the lines of the board with a for loop. However, in order to make this part of a game, I wanted a way to capture input events. SVG elements support an onclick() attribute which is a reference to a Javascript function. This is set in the same way as any other attribute.

While using onclick(), I used some other techniques. The first is to remember to set the pointer-events attribute (remember that I said that some SVG properties have a dash - this one cannot be set using the dot notation).

The second is that it is possible to have a hidden element that can still capture pointer events.

The third is that I set the id property for each pickable element and I built the id into the onclick() event as an argument. This means that each pickable element can identify itself when selected.


var circProperties = {'visibility': 'hidden', 'pointer-events': 'all', 'r': '10'};
circProperties.id = 'circ_10_10';
circProperties.cx = 10;
circProperties.cy = 10;
circProperties.onclick =  'playMove("'+circProperties.id+'");';
createSVGElement( 'circle', circProperties);

In my real use of this example, the cx and cy and id properties are set from loop variables.

Unfortunately, SVG elements do not have drag events. A simple click event is fine for Go but less than ideal for some other games. For example, in order to implement a game of Chess where the obvious user action would be to select a piece and drag it to its new position, I would need to select a piece and then select a new position. This requires some state in the Javascript and is less obvious for users. One thought is that, when a piece is selected, I could highlight the legal positions that it could move to. This requires the game logic to be embedded in the Javascript rather than at the server side but it would not be too hard.

Alternative Approaches

As I mentioned above, I did this work in Firefox. I like Firefox but I found that SVG support in IE and Konqueror is very limited (i.e. I couldn't make any of this work but I didn't try for more than a few minutes). However, A comment to my last post from Brad Neuberg pointed out that the SVGWeb project (http://code.google.com/p/svgweb) should add SVG support to IE and other browsers. I have not had time to try this but I will definitely give it a go.

However, I have also considered other approaches. One of these is to generate a picture file and lay it over an image map in straight HTML. This requires generating the complete picture at the server side and then sending the picture file to the browser. This picture file will definitely be larger than fragments of XML containing move information and the server will actually have to generate the picture. I have dabbled with generating a PNG file from Python and it can be optimized quite heavily but it is still less efficient than using SVG. However, for compatibility reasons and to see just how difficult it is, I will probably try it out at some time.

Thoughts on a Browser-based board-game

2009-08-10T19:16:00.004Z

One of my spare-time projects over the last few weeks has been a browser-based, pure HTML and SVG implementation of the Go board game. This has reached a usable state (although definitely still pre-alpha) and is worth some reflection on the architectural implicatons.

My starting point was that I wanted to build the game using a Django / Python back-end and HTML / AJAX / SVG front-end. The project came out of my experimenting with Django and I am considering other types of game but Go was a good starting point.

A more commercially realistic approach would be to build a Java or Flash client but I wanted to see how much could be achieved using SVG. Also, I dislike Flash (due to its over-use) and I am tired of problems with Java installers.

Client Implementation
The client worked fine once I read up slightly on Javascript. Directly creating SVG objects has a namespace quirk but otherwise works very easily. I chose Go because the graphics are easy and the game-play just involves selecting points. My experiments have shown that it would be straightforward to use graphics files for the board and pieces if I wanted a more attractive appearance.

The two drawbacks that I have found with this approach are browser support and the lack of drag and drop.

My early experiments with IE and Konqueror show poor SVG support so I just worked in Firefox for now. If I persist with the project longer term then an alternative would be to draw the board and pieces server-side into a PNG file and send that combined with a client-side image map. This would be more work on the server side and would require more data transfer but I would quite enjoy creating a basic PNG library in Python.

SVG includes onClick events so I was able to put hidden items on the points to be selected. This works fine for Go but would not work in, for example, chess where a drag and drop action would be desirable. It would be possible to click once to seect the piece and then again to select the destination but I will have to try it out to see if it feels too clumsy.

Server Interface

I was deliberately working with a Django back-end intended for ease of deploymen so I wanted no non-standard server features. this proved interesting as what I initially considered to be a clumsy architecture turns out to have some attractive aspects.

Dedicated server versus HTTP transactions

If I had not been basing the design on Django then I would have started with a server that kept each active game in memory and applied moves as they were received. As I was using Django, each browser action was totally separate and so the POST request to play a move requires the game to be loaded from the database, the move applied and a response sent to the browser with captured stones etc. and the game saved in an updated form.

In a quick-moving game, having to load and save the game for each move is an overhead but when a longer-term view is taken the Django approach (really , the generic web-server approach) has robustness advantages. With a server running direct connections to clients I would have to work out what to do if a connection went down - how could a client re-connect and resume their game. Also, what if players want to pause a game overnight? Basing the whole approach on an atomic move basis automatically handles these aspects and also provides a re-play feature almost for free.

Server Push

Another aspect that was less convenient was trying to implement server-push. I want to be able to update player A's browser when player B makes a move (I also included a chat pane in the game which has the same requirement). In a dedicated server, I would use a long-standing read from the client and would provide a practically instant response. This is not really workable using an HTTP server.

In theory, I could use a dedicated HTTP GET request to fetch updates and have the server delay replying until there is event data to return. On the browser side this should work fine. Unfortunately, it has two problems on the server side

Firstly, it means that each game would involve two dedicated requests outstanding at all times in addition to the transient requests to fetch game state, post moves etc. With conventional HTTP servers this would likely cause a scalability problem as each connection requires resources and servers are configured to limit connections.

Secondly, HTTP servers running Django or other CGI frameworks keep each request separate (for very good reasons). Therefore, if I had a pending request for player A and a separate POST from player B it would be tricky to communicate between those requests. This might not be insuperable (and would be trivial in a dedicated server) but I did not investigate further given the resource issues.

This means that my event handling is based on poll-based requests that mostly return no new event data. This is fine in terms of server resources but it does consume network bandwidth and means that the responses can be delayed for a few seconds. This type of problem has been met by a lot of other people (look up Comet in Wikipedia for example) and I expect that I will return to it to investigate further at some time. For example, at what point does the server load from repeated requests with no events outweigh the resource cost from outstanding requests and how can I pass events from one request to another (maybe the new Django signals will help)?

Implementation Efficiency

One lesson from this project so far is the ease of use of Django. Because Django takes care of all of the plumbing, I have 200-300 lines of Python to implement the rules of Go, about 100 lines of plumbing and chat code in Python and about 200 lines of Javascript to draw the game and handle moves. The whole project took less than a week of evenings so far and that included a chunk of learning. Adding more error checking and more attractive admin interfaces will probably double the size but the whole project is easily understandable and manageable.

Why I think Opera Unite is missing the point

2009-06-17T20:31:00.002Z

I watched the PR fuss over Opera Unite with some interest but I do think that they have missed the point. This posting by Chris Messina raises some valid points about the use of Opera as a proxy but I think that this covers only some of the weaknesses.

Over the years, I have seen proposals for servers running on individual personal computers and on mobile phones and they consistently look to me like a solution looking for a problem.

Technically, it is easy to create or configure an HTTP server on most devices. On a PC, a server like Apache is not hard (for a techie) to install and configure although I don't see average users trying it. It is not hard to port Apache to smaller platforms but I have also seen (or created) custom HTTP servers developed for more embedded platforms. Creating a basic HTTP server in C/C++ or Python (or other languages) is easy. All of the clever options that 'real' web servers support are more challenging but probably not required for smaller devices.

What is clearly more challenging is deployment and configuration for average users, but this ties into the user cases. Given the assumption that it is practical to create an HTTP server on a range of devices, the question arises of what to use it for and how is the address made accessible. I think that both of these are real problems. I think that Unite addresses the deployment and configuration challenges but still does not come up with good use cases.

Opera try to handle publishing addresses by means of their own proxies. The comments in the blog above show some of the problems with this but I don't see a good alternative. Unlike standard web servers that have relatively stable IP addresses and that are hooked into the DNS system to make URLs usable, personal computers and mobile devices normally do not have fixed IP addresses. They commonly use DHCP to get a transient IP address. This means that it is difficult for a remote client to know what address to connect to. Also, PCs and mobile devices are not necessarily on-line the whole time (but more of this below). In practice, a personal or mobile HTTP server would need to get an address and then push its address to some central addressing server so that clients could find the address and tell if the server is online.

When we come to consider what should be published, I think that Opera are just wrong in their assumptions. I think that it is daft to try to publish static content such as images or other media files from a PC or mobile device compared to using a hosting service. If a user wants to self-publish then they have to consider issues such as backups and access restrictions. I seriously doubt that most users will want to do this. Why not just push the content to Flickr, Facebook or one of the groups services? Even if you don't fully trust the hosting services' backups they will provide basic access control and the content will be available approximately 24/7. If you want to retain control over the content then set up a cheap hosted web site. If you want to exercise access control then consider email or set up a proper site! If these files are made available from your home PC then you will have to effectivekly upload them whenever they are accessed - most home accounts are not great at uploading so this looks like a bad idea to me.

If you do want to publish any static content then surely you want the content to be available most of the time. I don't know about you but my laptop and smartphone are not connected 24/7. They are offline or turned off for substantial parts of each day.

The Fridge and Lounge services in Unite look a bit me-too to me. I could knock up such a service on a hosted site very quickly and they would be available when my home PC was turned off. I haven't looked but I would be very surprised if this type of service was not cheaply available with various hosting services. I guess that Unite makes it available for free and with less work than setting up a hosting service - you trade increased convenience for the price of going through Opera's servers. As an alternative, try setting up a closed Yahoo group and you can get many of the benefits.

I do see value in a local HTTP server but for more specialised purposes:

If a device (maybe embedded or mobile) wants to make its status or other dynamic information available then HTTP is a good enough protocol to use as it is well understood. I could see the value in a service wherby such devices ran a server and then informed a central service that they were available for information retrieval. This is a pretty closed use case.
Specialised HTTP servers such as synchronisation servers are useful but they will tend to run when required and have specialised methods for making contact with a client (you can tell that I used to work on OMA Datasync).

So, although I like the idea of running a local HTTP server on a PC or mobile device (it is quite fun to consider the porting issues), I still think it is a solution looking for a problem. The Opera site mentions future developments and the ability for developers to work with it but, heck, if I want to create web services, I will play with a hosted service. I can use Django (just one of many) to build something and all that Units gives me is some convenience at the cost of beling locked into Unite. I would love to be wrong but I don't think I am.

P.S.
Having mentioned Chris Messina's post above, I subsequently came across a response from Lawrence Eng here.

I should add that I am not criticising Opera for not going open source
and I am not commenting on the technical implementation - I have not
spent the time to look at that so I am prepared to assume that it is
fine. My view is that the whole concept is a problem looking fr a
solution. Some of the blog comments make the point that it is easy
for naive users to use and this may turn out to be sufficient.
Alternatively, somebody may come up with the killer use case that I
have overlooked. until then, I will rem ain interested but sceptical.

Further thoughts on implementing non DBMS storage

2009-05-17T16:48:00.001Z

Since my last post I spent a couple of evenings putting together a pure Python storage class to store arbitrary chunks of data in a way inspired by Haystack. This proved quite easy in Python (unsurprisingly). It would also be quite easy to implement in in C/C++ for performance but I am not convinced that is necessary and a pure Python implementation has the virtue of being simpler to deploy.

The next stage is to integrate it into Django so that it can be used as a model field and extend the admin application and generic views handle it. This should be almost as easy so I am now reading the nice tutorial on creating new model fields along with the FileField source code.

Interestingly, I came across this story via Reddit about drawbacks of CouchDB.
http://blog.woobling.org/2009/05/why-i-dont-use-couchdb.html
I had already decided that I wasn't keen to play with CouchDB at present because I think some things (such as authentication and real regular data) are more efficiently done in a normal SQL DBMS.

I am now brooding about creating a CMS with a combination of my storage classes and search but I am trying to work out if it is any different to a Wiki with search facilities.

Some thoughts on a text storage system for Django inspired by Facebook's Haystack

2009-05-11T17:39:00.002Z

I have been looking at Django lately as I wanted a project to play with and I thought that I might extend it in some way. In some ways, Django has been a disappointment as it is quite mature and very powerful so the opportunities for tinkering that I wanted are not really present. This makes it great for real web developers but less great for me personally ;-)

However, one aspect that kept nagging at me was the use of DBMS fields for large quantities of text and for searching (this is not only a feature of Django, of course). When I used real DBMS (back in the nineties) we had to be very careful about optimising column sizes and we would not have dreamt of storing whole blog entries in a database, let alone larger bodies of text. I know that hardware has gotten cheaper and DBMS have improved but this still feels to me like using the wrong tool - as if the designers used a DBMS for everything because it was the tool that they knew best.

I was browsing for search software and found examples such as Xapian and Sphinx. Again, I had used a similar tool in the late nineties when building an in-house knowledge database but the world has moved on significantly since then. I installed Xapian on my Linux laptop and was shocked by how easy and quick it was to feed in text and search on it. I fed in a Robert E Howard novel using the Python binding and didn't notice it go in. A search was also extermely fast. This provoked the thought of extending Django with Models that include searchable fields. It should be possible to simply tag model fields as searchable and have a Django extension automatically index them with Xapian (or another search tool - the choice would be transparent) and then extend the generic views to include searches. This feels like a project that would be technically interesting and that would extend Django in a style that is consistent with Django.

One issue that occurred to me when thinking about Xapian was the sequencing of index updates. I am wary of indexing as part of the data creation in a web server. If multiple users are creating data simultaneously then Xapian would not like having too many simultaneous updates. As Django is deployed with a range of web servers (or certainly with a range of interfaces to web servers) I would be reluctant to try to implement locking across multiple requests. My current proposed solutuion is to create an internal message whenever an item is added that needs indexing and then to pick them up and deal with them in a batch from a specialised request. Django provides such a messaging system so it is not necessary to add another table for the purpose and the indexing requests could be kicked off by a cron job or manually by a sysadmin. The use of a single batched indexing systems allows more efficient indexing sessions.

While considering non relational DBMS storage systems (I browsed through articles on BigTable and CouchDB), I came across another story that I noticed on Reddit about Haystack - the storage system used by Facebook to store photographs. Apparently, Facebook uses MySQL extensively but the sheer volume of photo data made it impractical for photo storage. Haystack is used to store phots in a very efficient manner - the Haystacks contain meta data pointing to the photo data; all data is appended to files so multiple reads can take place while appending takes place; the data is supposed to be structured so that it can be retrieved with minimal disk seeks.

Putting these ideas together, I thought of a text version of Haystack. Each text object can be appended to a filing set and the database can just contain its meta data (file name, offset and size). This makes the relational database smaller and so more efficient. The text can be indexed for searching as it gets saved and it should be possible to extend the generic views to handle this transparently. Other ideas from Haystack and CouchDB that can be applied include versioning and never deleting content. I would probably also have a configurable maximum size for a storage file and have the system just keep extending the system with new storage files - I have a bias against unreasonably large individual files for backup and management reasons.

If I get a few hours, I should be able to prototype this and feed in a range of literature from project Gutenberg as a test set.

Use of the Broswer as a (cross-platform) UI

2009-05-03T16:03:00.001Z

I was considering some mobile application development and was debating which UI library to use. Some thinking came up with the idea of using the browser as the UI with a local HTTP server. This is really suitable for creating PC local applications rather than mobile apps at this time but more powerful mobile phones may change this.

An obvious attraction is that it helps the separation of content from presentation (as long as you do not over-indulge in Javascript, let alone do anything daft such as using Java Applets or Flash)

Browser as UI

With modern browsers and straightorward applications, a perfectly good UI can be built. It is not really possible to create a very complex UI but I claim that most applications do not need a very complex UI - just how dynamic and immersive does a PIM application need to be?

Applications such as Gmail and online office packages demonstrate what can be done.

HTTP server accessing local functions

Using an HTTP server for the 'back-end' functionality means that any language can be used - there really is no need for anything in common with the UI. My personal preference is Python so I am going to experiment with Django.

Using HTTP as the interface can be seen as overkill or inefficient. Rather than reading data directly from the UI and modifying it directly, everything has to be serialised and transferred over an internal socket. This is true but the overhead is probably negligible compared with the actual application functionality. The real queston is whether it provides a convenient programming environment.

Most web development is concerned with preventing insecure access but if the local HTTP server is the application engine then it needs full access to the local machine - not really a problem.

The real issue is deployment of the server - no normal user is going to install and configure Apache and MySQL just for one application. The good news is that the server does not need to be scaleable as it will only be serving one user. This means that lighter-weight servers such as Django's development server or one custom-built from Python's simple HTTP server classes can be adequate.

Challenging your own views

2007-11-22T08:20:00.000Z

Yesterday, I read this interview on The Register with Adam Curtis about how mainstream media are failing to take a lead by paying too much attention to bloggers.

It is quite long (as these things go) but it provoked several trains of thought in me that were not directly the subject of the piece.

One of the themes in the article is the existence of different groups (of bloggers) with fixed viewpoints who refuse to listen to any opposing or alternative viewpoints. The article takes this as a reason why the mainstream media (such as the BBC) choose to tread a middle path between opposing groups rather than trying to form their own view.

My personal lesson from this is the weakness of being too blinkered in ones views. If you hold strong views then it is very easy to keep reinforcing them but in a technical and management arena it is useful to chalenge your own views occasionally. I am in some ways an archetypical 'Guardian' reader and I can be comfortable reading newspaper articles and columnists that agree with me but too much of that breeds complacency so I make a point of reading 'The Independent' quite often to get some opinions that I don't automatically agree with. I have tried reading newpapers with views further away from mine and it is much more difficult to critically assess their views - get too far to the right and I just get angry...

In technical areas, I often take part in discussions with engineers puttng forwards views with which I disagree. In some cases, their view is not that controversial but it is simply not what I wanted to hear because I came to the dscussion with a proposed solution of my own. In other cases, I really do disagree with their view but I make a point of hearing them out and trying to see the value in it, despite my biases. This can require some emotional effort to be patient and objective, particularly when we are working under time pressure or when the engineer presenting his views is doing so clumsily, but I find enough occasions when there is value in the other point of view to keep making the effort.

Why do Managers behave like idiots and why are Engineers so obstructive ?

2007-10-24T18:23:00.000Z

All engineers have anecdotes about when one or other manager behaves in an idiotic manner, making decisions that are obviously stupid. Often, the more senior the manager the more frequent the bad decisions and the more serious the outcome.

Alongside this, all managers know that getting engineers to do something new or different can be frustrating to the point of apoplexy. Engineers seem to have an innate desire (and skill) for finding objections and reasons why a proposal will not work.

I suggest that both of these problems are caused by the different environments and needs of managers and engineers. My starting point is to acknowledge that both managers and engineers tend to be highly intelligent and highly motivated – neither group is stupid or deliberately obstructive.

Managers have two problems that are relevant. The first is the scope of information that they need to handle. The second is the effect of friction in any organisation.

By information scope, I mean the range of information for which a manager is responsible. Even a quite junior manager is likely to responsible for a number of projects and the more senior the manager the wider the breadth of the organisation that they are supposed to know all about. This means that a manager cannot possibly know everything about all the projects etc. under their control. This does not stop more senior managers expecting instant answers to random and detailed questions. When necessary, a manager can learn all about one area (and engineers may be surprised at the speed at which a good manager can filter and absorb information) but they cannot learn everything and maintain that knowledge.

The information scope problem means that a manager will tend to have to make decisions based on partial information. The more senior managers get their information from their subordinates so their information is second-hand as well as partial.

The concept of friction comes from Clausewitz and refers in this case to all the factors that make even the simplest organisation task difficult. Clausewitz ascribed friction to the physical danger of war and the physical challenges. Most organisations do not have this level of danger but communications difficulties, conflicting priorities and natural resistance to change all make it unexpectedly difficult to carry out projects or to make changes. This means that any successful manager has to be a highly motivated problem solver. One aspect of such behaviour is an ability not to be put off by minor obstacles. If a manager stops to re-think the plan whenever somebody objects then they will never succeed. In contrast, if they ignore or override objections they may hit problems but they have a better overall chance of success. Of course, the really good managers have to decide which objections are serious and which can be safely ignored.

One side effect of the information scope and friction problems is that managers have to be able to look at the big picture without getting bogged down in minor details. Oh, and they are commonly under time pressure which can make them impatient with unnecessary details.

Turning to obstructive engineers, we can see that a good engineer needs to care about detail and needs to learn everything relevant about their subject or immediate problem. As engineers tend to work directly on their problem, they may be less affected by organisational friction (apart from the friction in getting budget or resources) – they tend to be hands-on.

When engineers discuss technical problems, they need to have a shared knowledge base that can be detailed. Therefore, they may have to provide a large amount of background information in order to present a problem or ask a question.

If an engineer ignores an inconvenient problem for reasons of expediency then the problem is unlikely to go away. It is more likely to return when least expected but bigger and uglier than before. Therefore, engineers tend to focus on problems as they find them and are uncomfortable moving on until they are convinced that the problem is solved (or at least can be solved for a reasonable cost).

To a busy manager, engineers are people who keep bringing up problems, describe them in a long-winded way and need constant ‘motivation’ to keep them moving forwards.

To a conscientious engineer, managers are ignorant people who don’t have the patience to understand what is really going on and who prefer to deal with problems by ignoring them.

Thoughts on interviewing software engineers

2007-06-12T11:42:00.000Z

Over the weekend I came across this blog entry.
http://blog.pmarca.com/2007/06/how_to_hire_the.html

I do a lot of interviewing and I am very aware of how difficult it is to make an interview effective so I am always interested to read about interview techniques for software engineers.

Mostly, these focus on problem solving but I distrust these. I do have a collection of interview problems, some based firmly in software engineering and others more abstract, but I have tried them all myself and I don't believe that the ability to solve arbitrary problems under interview conditions is a good indicator of ability.

To be more accurate, I don't believe that the inability to solve one particular problem under interview conditions shows that an engineer is incompetent in general.

Code Generation 2007

2007-05-29T19:57:00.000Z

I had the pleasure of attending the Code Generation 2007 conference held at Homerton College, Cambridge on the weekend of 19 and 20 May (actually, it started on the Friday but I was unable to attend until the Saturday). I went because it was some years since I looked at code generation and I wanted to see how the state of the art had changed. The presenters were very competent, academically sound but grounded in real life and the conference was friendly and well run (thanks to Software Acumen).

I don't think that I'm going to be able to apply much in the way of code generation to my day job but it was a very well run conference and certainly thought provoking. I picked up a couple of books and I have dug out a couple more from my shelves.

I didn't find much new in the generic modelling and code generation techniques although it was good to have my views checked by comparison with current best practices and I will take a long look at how to preserve design intent at some time. Krzysztof Czarnecki of the Univ of Waterloo in Canada was notable in this area. The area that were new to me was the coverage of Domain Specific Models (DSM) and Domain Specific Languages (DSL).

Two good sessions on DSMs were run by Juha Pekka Tolvanen of MetaCase and I was particularly struck by the Symbian OS S60 application generator. Juha Pekka was very good at identifying the success factors for a DSM to avoid misplaced use.

Nice short article by Matt Stephens about individual differences

2007-02-12T13:02:00.000Z

The article is on The Register Developer site -
Non-Humans need not apply.

In it Matt makes a case for treating software engineers as individuals with unique skills and characteristics as opposed to the generic 'resources' that MS Project thinks they are.

I can also recommend reading up on Belbin's team roles, if only for the sake of self-awareness.

Estimating and why it is so hard

2007-02-11T11:40:00.000Z

In this essay I want to consider the topic of estimating in software development. As any software engineer knows, there is a constant pressure to provide estimates for how long a task will take and it is notoriously difficult to come up with decent (by which I mean reasonably accurate) estimates.

In order to understand this, I want to consider what an estimate is, what the limitations on an estimate are and to touch on some of the techniques that can help.

What is an estimate?
An estimate is a prediction of the future. The most common form of estimate required in software engineering is how long it will take to do a task. Depending on the context and the sophistication of the organization, the estimate may be in terms of man-days or elapsed time and may be loaded to some extent.

Why is estimating hard?
Predicting the future is hard in general. If it was easy then stock market analysts and astrologers would be out of business. In the field of software engineering, some tasks can be routine and some tasks are novel. The difference is crucial - if you treat all tasks as novel then it can be impossible to move forwards but if you treat all tasks as routine (often the assumption) then you fail to take into account the real difficulties and you are almost certain to come badly unstuck.

Routine tasks are those that we have done before and that we have a reasonable chance of successfully doing again. They are unlikely to be totally routine but the probability of surprises is low.

Novel tasks are those that involve a high level of creativity or problem solving. We don't know how to do them before we start and so our prediction of how much work will be involved or how risky they are is not likely to be accurate.

There is a host of techniques to help with estimating routine tasks. These have been analyzed extensively and are covered by books and training courses. It is still possible to make mistakes but the probability and the consequences are manageable with some planning.

There is a much smaller range of techniques to help with novel tasks but there is some hope.

Limitations to Estimating
A naive view of estimating is to expect an estimate to be a single figure that should be more or less accurate. For example, if an engineer estimates that it will take five days to program the Wurlitzer component then we might expect the Wurlitzer component to be complete five working days after he starts. if it takes six days then we will view that as a 20% over-run and so on.

The first catch is how well we have defined the task that has been estimated. Does the task include code review and rework? How about unit or integration testing? How about documentation? This type of problem can be overcome simply by taking pains to be clear about the task definition and the easiest way to achieve this is to work out (and publish) a set of organizational norms so that engineers and project managers know what they are talking about without having to argue about it or write lengthy project documents.

The second catch is is that a single figure gives no measure of the risk. If five days is the most likely time for completion then what is the probability of going over by one day, or two etc. A task that has a likely time of five days but a worst case of ten days is significantly different from one with a likely time of five days but a worst case of 20 days. A standard way to represent this is to use 3-part estimates. For each task, estimate the time taken in the best case (if nothing goes wrong), the likely case (assuming normal levels of disruption and problems) and the worst case (assuming lots of problems). As an aside, the real worst case is always infinite, what we are really after is a 95% estimate; a figure within which we expect to finish in 95% of cases.

There are formulae to combine the best, likely and worst case figures to give a single figure for planning purposes but I find the worst case estimate useful simply because of the thinking that it triggers. If you can get an engineer to avoid adding a standard percentage to all estimates to get the worst case then the interesting tasks are those for which the worst case is much larger than the likely case. Look at these tasks more closely to see if there are risks or uncertainties that could be eliminated.

The other value of 3-part estimates, apart from helping to identify risky tasks, is that it makes explicit that there is a margin of error on any estimate - it is not a guaranteed figure. It can be interesting to consider how the margin of error changes over time. Theoretically, the margin of error is linked to the degree of knowledge or uncertainty about a task

Initially, we have a very limited amount of knowledge about a task (certainly if it is novel). As we progress through the task (or through other tasks that affect it), we gain more relevant knowledge and the margin for error should decrease (when the task is complete the margin for error should be zero!). This is why I believe strongly in re-estimating during a project and a task to get better estimates as we go along.

Estimating routine tasks
I will not spend many words on this subject because there is such a rich body of work already available. The key techniques are to use historical data and check-lists and to identify causes of variability. Measures of task size such as function points or lines of code can be a valuable starting point and measures of variability such as engineer experience or the complexity of the other components to be interfaced with can be useful.

Even with routine tasks, estimates will never be 100% accurate. Unforeseen problems and variations can trip up the most experienced engineer. However, with some effort, the 3-point estimates can cover most of the tasks.

If a project includes a mixture of routine and novel tasks then it is worth minimizing the risks on the routine tasks just on principle but the majority of the uncertainty will come from the novel tasks.

Estimating novel tasks
With novel tasks, it is difficult to predict the work based on history so we need an alternative approach.

The only real way I know to approach a novel task is to do some up-front design. This is aimed at eliminating some of the areas of uncertainty and trying to break the task down into smaller parts that are more easily understood. It is important to understand that you need to do some work before you can start to come up with the estimate. It is not reasonable to ask for an estimate for a poorly understood task without allowing for some analysis.

The trick is to decide how much analysis is required before coming up with the estimate: too little and the estimate will be dangerously unreliable, too much and the work will never start.

Depending on the types of uncertainty, you might try prototyping but this does not replace the need for design work.

What is a Software Architect?

2007-01-28T18:11:00.000Z

This essay discusses the tasks of a Software Architect and what properties and skills an engineer has to have to fulfill the role.

The headline skills required are Design and Communication with some secondary Project Management skills.

Communication

A software architect has to be able to communicate with a wide range of people including customers, suppliers and partners as well as within the development team.

As in any situation, if you want to communicate effectively with somebody you need to understand their needs and their point of view. Therefore, the software architect needs to understand these different groups well in order to be successful.

This communication ability can be difficult for some software engineers to acquire, particularly if they have spent a lot of time in a back-office or working in specialized technology. The good news for such engineers is that this is the standard stuff of management training and these skills are well worth acquiring in their own right.

There are several reasons why a software architect needs to communicate with various groups:

the architect needs to talk to the customers to find out their use cases and their requirements. This also requires that the architect be able to understand their business so more management skills come into play.
The architect needs to present a proposed solution to customers. Even if there are sales or marketing professionals involved, the architect will be needed to answer technical questions and he or she may end up as the external face of the project or system.
The architect needs to communicate within the team to ensure that everybody understands the overall design and what is needed.

Representing the chosen solution in this way requires that the architect develop good 'people skills'. An engineer who is not a 'people person' will struggle to be a good software architect.

Design

It is probably obvious that a software architect needs to be skilled at software design but it is still worth stating. It is also useful to consider the additional design skills that an architect needs that designers of smaller components may not have developed.

The principal difference is in the scope of the design. Almost by definition, a software architect needs to design complete systems that may consist of a number of parts that communicate or interface with each other.

The architectural design must take into account overall performance and robustness rather than focusing on one part in isolation. This kind of overall optimization is more difficult than purely local optimization and may require some parts to be less optimized for the sake of the larger picture.

The architectural design must be heavily concerned with the split into sub-systems or components and with the communication between them. Therefore, an architect must be good at defining or finding the interfaces or protocols between the parts of the system. In general, if there is a good (or good enough) standard or protocol then it should be used rather than a newly invented one. This means that a software architect needs to be familiar with all the different protocols that may be useful.

Although the software architect is concerned with the 'big picture' design, he or she has also to be aware of the lower level design issues. This is comparable with a civil engineering architect needing to be familiar with the properties of concrete, steel, brick and glass in order to know how best to use them. If a software architect is not capable of designing the building blocks of the overall design then their design is likely to be flawed in detail (or they will need a lot of support from local experts).

This is why I think that a software architect needs to find ways of keeping their technical skills up to date. Where possible, some prototyping or hobby programming contributes to this but extensive reading is also required.

Making sure that the architectural design is of a high quality in the first place is critical but the architect also needs to make sure that the quality of the actual deliveries is maintained. This is not really a design issue but it is a technical issue and the architect is likely to be the final arbiter on whether the quality is good enough.

Project Management

In many cases the software architect will not also be the Project Manager but the architect needs to have a good awareness of project management issues and must be committed to supporting the project manager. Of course, if the architect is also the project manager (maybe with a title such as Software Lead) then the need is even stronger.

The architect needs to consider development and maintenance costs for the different parts of the system. If there are design choices then the costs are a factor alongside technical suitability.

When the project plan is under development, the dependencies and ordering of tasks must be based on technical considerations. The choice of which components to implement first (or which parts should be designed and implemented in parallel) may be based on technical dependencies or on which parts can be tested in isolation or on which parts carry the highest risk. All of these are technical questions that the architect must answer.

Once development is underway, risks must be kept under review and the project manager needs the architect to decide what is happening to the technical risks.

Within the project team, the architect is best placed to make sure that everyone knows what is going on from a technical point of view and he or she is responsible for contributing to team morale (in a positive way!). An architect who is a good leader and is respected by the team can make a tremendous difference to the success of a project. One who fails in these regards is a serious danger.

What is the point of a design?

2007-01-15T21:23:00.001Z

This essay explores the purpose of a design. This begins to show how multi-faceted is the term design. As a verb it refers to making plans or creating something; as a noun it refers to the plans, sketches etc. (for the sake of convenience I ignore the other meanings that don't really relate to software). The design process is the creative act but I want to focus here on the design artifacts - what might be called the design document, whatever form it takes.

I suggest that there are at least five uses for design artifacts and that these uses correspond to stages in a design and development process.

Design as a thinking aid

The first use of a design is as a thinking aid. Early in the design process, the designer is engaged in the pure creative act (at least that's what I claim when I am discovered staring out of the window). Although the earliest stages may take place solely in the head of the designer, some drawn or written form will normally appear quite soon. As a minimum, I will draw some form of picture just to stare at - it provides a structure on which to hang my thoughts.

With any design of real complexity there will be one or more difficult problems to solve or critical aspects to handle. Exactly what these are depends on the system but common ones include performance, robustness or complex algorithms. While attempting to solve these problems, the designer will want to draw pictures or create state or sequence diagrams to explore them and to test theories. If the problems are suitable then a partial prototype can be very useful.

At this stage, the design artifact should contain only what is needed to investigate the key problems: obvious or easy parts of the design can be taken for granted. There is no need to have a standard form or to cover all aspects of the design and the design does not have to be optimized for communication.

Design to support Peer Review

The second use of a design is to support a peer review of the design. Actually, I want to include both formal peer reviews and informal design discussions but I will cover the formal reviews first.

In order to support a peer review, the design artifact needs to support communication - the design concept needs to be shared with the reviewers. This is where the use of some standard or shared notation kicks-in to minimize the learning effort.

It is likely that a design for peer review will need to include at least a minimum of standard information to provide context but the emphasis should be on describing the difficult or critical aspects so that they can be thoroughly reviewed.

Of course, there is a hidden gotcha here. If the designer skimps on the detail for the parts that can be taken for granted and is wrong in their judgment then the peer review will miss these problems. This is an argument for covering all aspects of the design to some level of detail but I still believe that more effort should be expended on the aspects that the designer knows are problematical.

Because the purpose is to allow Peer review, this type of design must include more than just static pictures. State diagrams or sequence diagrams should be used to show how the design works or how it can be used.

By separating the design as a thinking aid from the design for peer review I have omitted an intermediate stage, the design as a tool for early design discussions. Wherever possible, I like to get input from other people on a design and this requires something to talk about. If you are lucky, this can be a picture drawn on a whiteboard with minimal effort. If the design is more complex (or at least the part of the design that you want to discuss is more complex) then you may need to create a more formal representation in order to get the other contributor(s) up to speed. Whichever form you choose, this is not a formal peer review so you can focus on the aspects that you want to discuss without filling in the rest. As this type of discussion is cheap to carry out, I recommend them early and often.

Design to define and communicate structure & interfaces

Any system of more than minimal complexity will need to be split into parts (I avoid using a term such as components because it can have specific meanings in some methodologies) for understanding, implementation and, probably, operation. Exactly what form the decomposition takes depends on the system. A Web services architecture will involve distributed processes, each of which may be further decomposed, while a stand-alone application may be decomposed into classes or DLLs.

The decomposition of the system and the definition of the interfaces is crucial to the design of a system and so a design should describe them. This type of design can then have several uses. It can be used by teams or individuals to implement the parts. It can be used by testers to implement various levels of tests. It can be used by customers of the system as documentation for using it.

Although the decomposition is essentially a static picture of the system, the interfaces will almost certainly benefit from a dynamic description such as sequence diagrams. For those interfaces that will have external customers, I like to have a draft tutorial document at this stage, even before coding starts.

With this type of design, the internal structure of the parts does not need to be described unless they are critical or difficult.

Design as a work description

I dislike the development process that this type of design implies. I have seen designs created to guide 'less skilled' coders but I would much rather encourage the implementers to create any additional designs that they need where the internal structure or behaviour of components is not obvious and then carry out early and possibly repeated reviews. In this way, the implementers grow in skill and ability and there is no artificial divide between designers and coders.

This approach does require more communication and reviews within the team but I regard that as beneficial.

Design as a maintenance guide

At this point I have jumped from design artifacts created before or as part of implementation to design artifacts required after implementation. I think that this aspect is often overlooked or misunderstood and I will expand this view below.

If a system is to have a significant lifetime then it will need to be maintained. Any designer who believes that their system will never need maintenance is lying, either to themselves or to others and maintenance can include fixing defects and extending the system for new uses. Given the likely need for maintenance it is sensible to consider what type of documentation will support it.

During maintenance we can assume that we have some other artifacts - at least the code for the system and, with luck, some working test cases. Maintaining design documents after implementation is always problematical; under time pressure, it is always tempting not to bother to update them. Therefore, we should try to keep maintenance support to the useful minimum.

If your code is well documented (hint - Doxygen at least) then you do not need to reproduce detailed API documentation but maintenance documentation should include

- an architectural overview
- interface descriptions where in-code comments are not adequate (keep the sequence diagrams)
- records of interesting design decisions (to avoid maintainers inadvertently undoing them)
- guides to expected extensions

Summary

The key point overall is not to assume that there is one true form for a design document and it must be created early and then maintained throughout the design process and system lifecycle. Instead, we should consider what the purpose of the design document is at each point in time and set the content accordingly.