Latest updates to Discovery – August 2012

Following user testing and feedback, Discovery has been updated to include new features and services.

Newly added:

• Home Guard records for Durham – descriptions, images and new process for ordering closed records

You can now:

• save your search query by copying the url, to add to emails and web pages
• bookmark pages and searches

We also have a new advanced search form, which allows you to:

• separate reference search from former reference search
• search for up to three references
• search by date of opening
• search by closure status

We’ve added new filters on the search results page:

• by date of opening
• by closure status

A number of bugs in the system have also been fixed, so you can:

• search by former reference
• download search results as CSV, HTML and XML files
• search with numbers, which no longer automatically set up an “OR” search
• search for alpha-numeric terms which are not references as keywords, i.e. MI5, MI6

Your feedback

We are releasing improvements as they are developed, rather than all at once, so we can continue to get your feedback and use it to further develop and refine this new service.

Each new release is prioritised based on your feedback and feature requests so please keep using Discovery and giving us feedback!

We need your help to develop and improve the system, so please let us know through the Comments box below what you think and what you would like to see.

You can also tweet your comments @UkNatArchives with the hashtag #tnadiscovery.

Have you ever wondered what happened to those departments that suddenly disappeared years ago? Or perhaps you are trying to find out which department took over the work of ‘Department [X]’?

The aim of this project is to represent changes to the names and functions of government organisations, or departments, which administer the state. These organisations are often referred to as the ‘Machinery of Government’.

In an effort to visually represent Machinery of Government changes, we have constructed the first of a series of timelines. Each timeline will be based on an area of government responsibility and will show the creation, closure and transfer of function between central government departments. The first of the series relates to foreign affairs, and the next will be defence.

Why we created this tool

The National Archives holds a wealth of knowledge about government function and structure, which is ready to be exploited to clarify something that can be confusing. Discovery is being developed to make finding our records easier than ever before and this visualisation provides links to it where possible so that users can find out more about each department and discover the records we hold.

We had also already encoded much of the data to support the innovative Semantic Knowledge Base (SKB) project for the UK Government Web Archive, which uses groundbreaking technology to provide linkages between concepts and help users find resources in the web archive.

Read more in our blogpost.

How to use the tool

We have provided a key within the timeline, which explains the symbols used. There are dates and names of Prime Ministers alongside them to provide historical context at each change.

The timeline provides an accurate overview of changes to government structure over time. However, users should be aware that it:

- shows ministerial departments only, not their associated agencies and public bodies
- covers only the period from 1945 to present day
- will be extended to provide links to more precise information about each department at the time and, where possible, to the SKB

The machinery of government data can be accessed as the timeline and in table form coded in XML.

Representing this complex information in an engaging way was always going to be a challenge. We hope that you find this useful and interesting and we welcome your comments.

Our first release includes a wide range of images taken from our records, including the Dixon-Scott photographs of Britain (1926 – 1948), the National Coal Board’s collection of  photographs, a selection of images of properties in London owned by the Crown Estate (1878 – 1967) and copies of some of the earliest maps held by The National Archives.

You can also use the map to identify the catalogue references to tithe maps, surrenders of monasteries, 19th century applications for building grants for schools and elementary school digest files (1854 – 1984). So, a search for Chertsey would reveal a copy of a medieval map of Chertsey Abbey, the catalogue entry for the surrender of the Abbey to Henry VIII, two photographs of Chertsey bridge in the 1930s and the catalogue entry for the 1844 tithe map.

To use the tool, select which collections you would like to search, then enter in the name of the place. Finally, set the range – the tool searches for records within a circle centred on the place you have chosen. The larger the range, the more places are included. Click on search and the map of the place you have chosen will be displayed with the symbols for any records or images which relate to that area. Clicking on the symbol will either link directly to the image or to the catalogue entry.

Please let us know what you think.

At the close of the event, 11 teams opted to present. They were:

• Jonathan Tweed and Kai En Ong (with Michael Smethurst, Faith Mowbray and Paul Rissen) produced the winning hack, a website using the @ukwarcabinet twitter feed to help support the exploration of wartime Cabinet Papers.
• Tim Hodson chose to work with SC 1, our collection of “Ancient Correspondence” and his visualisation of medieval power politics won joint second prize.
• Jamie Mahoney also won second prize by producing Show Me the Money, using details of Medieval lenders and debtors in C131. The tool mapped the financial relationships between the individuals featured and included a crowdsourcing tool to improve its own accuracy.
• Crystal and Steven Hirschorn won third prize with a number of visualisations of Olympic data scraped from Wikipedia infoboxes.
• Aleks Drozdov demoed an iPhone app for the Discovery API. This system will likely be deployed in the next year.
• Gwyn Jones and Vitaliy Oliynyk built DiscoVERY Lite, a new interface for interacting with Discovery.
• Peter Camfield worked with our “machinery of government” data to visualise the merging and splitting of government departments over time.
• Sam Mbale used Google Fusion Tables to geoference the ARCHON directory of UK and international archives.
• Jo Pugh georeferenced the Victorian and Edwardian photographers described in the catalogued portion of COPY 1.
• Jannette Mensch worked with the data in HO 144 and gave a presentation where she outlined some of difficulties in working with and interpreting the material. Read her blog on Hack the Record.
• John Cummings continued his quest to find an easy to use online tool for generating online map-based trails with embedded content.

Additionally,

• Matthew Pearce dug into data from dissolved companies in BT 31. Apparently there’s gold in there and he has set about finding it.

We want to thank all who attended, presented and helped to run the event.

We will be opening up the archives to web developers and designers to encourage them to gather and exchange ideas and make use of The National Archives’ data and documents.

We want to work with developers who share our interest in history, government and culture, to devise and build systems using our records, as well as datasets from across the web. Non-developers are also welcome – anyone with ideas for interesting tools, visualisations or systems developers could build using our data.

On the day, we will have records experts from The National Archives on hand, together with colleagues from the British Museum, British Library and Wikimedia UK to talk about their data and to pitch ideas.

Hack days at other institutions in the past have generated ideas including an online Domesday book, a detailed record of the Blitz and a range of apps and games.

Sign up for the event, taking place at The National Archives, Kew. The event is free but places are limited.

Follow the conversation on twitter: #hackon12

Datasets

We will be publishing a list of datasets that we will continue to add to in the weeks leading up to the event.

If you have specific requests for The National Archives’ data, email us at education@nationalarchives.gsi.gov.uk. We’re keen to discuss participants’ requirements and can arrange some limited digitisation in advance of the event.
 

The National Archives’ data

The National Register of Archives

A dataset comprising details of all the collections in the NRA relating to women has been published as linked data by Kasabi.

The UK Government Web Archive API

Access to this API will be made available to attendees over the weekend.

Archon

This directory of UK and some international archives is now available as linked data with a SPARQL endpoint hosted by Kasabi.

Discovery 

Discovery is The National Archives’ new catalogue.

Go to the API information for Discovery.

Legislation.gov.uk

Legislation.gov.uk is the official home of UK legislation 1267 to present.

Go to the developer zone for the API information.

Flickr

The National Archives’ photostream on Flickr contains over 5,000 images.

These can be reused via the Flickr API.

Labs

There is an additional list of datasets on Labs for use.
 

Non-National Archives’ data

AIM 25

Searchable archive collections in the London area.

Raw data is available.

Open Domesday

Anna Powell-Smith’s brilliant work with the Domesday Book.

A RESTful API is available.

Museum APIs

Museum APIs includes the British Museum and Portable Antiquities Scheme, National Gallery, Museum of London and many more.

Other useful APIs relating to museums and galleries are also available.

The National Archives has designed a RESTful interface that allows developers to query the search engine and our Catalogue database within the Discovery service application programmatically, and returns results in XML for further processing.

The service is offered as a beta with some functionality still to be developed.  If you would like access to our API, please email us at Discovery@nationalarchives.gsi.gov.uk, and please include the IP address from which you will be sending requests to our API.

Find out more about our API.

We will continue to add more datasets to this post, so do revisit Labs for new data to use. We will also update the data currently available, so please check the date in which the data has been added for the latest version. All the information about the data is below. Do let us know what you’ve done with the data by posting a comment on Labs - it would be great to hear from you.  To access all the datasets, go to: data.nationalarchives.gov.uk/records/.

The data we have made available through Labs is licensed under the Open Government Licence.

A lot of very clever developers produced an extraordinary range of hacks over the 24 hour event – you can read about some of them here.

I’m not any sort of hacker, I was there to talk about history. But it seemed churlish not to at least attempt to play around with some National Archives data and so I put together a simple (and very rough) visualisation using IBM’s Many Eyes platform, which I called Popethink.

Popethink takes all of the catalogue data from SC 7, our collection of Papal Bulls.  This is nothing to do with livestock: a bull is a piece of official correspondence from the papacy which takes its name from the metal seal (a ‘bulla’) attached to the bottom. We hold over 1,200 of these letters from Pope Innocent II in the 1130s up to Henry VIII’s break with Rome 400 years later. They have been helpfully catalogued into various types, some of them fairly obvious like orders, requests and exhortations and some a bit more arcane, like indults which allow certain named individuals to break specific items of church law. Each entry in the catalogue begins with one of these ‘key’ words.

Many Eyes can show these items as a word tree, showing all the entries branching off from successive words in the catalogue entry and this can show us very quickly who are the subjects of the Pope’s correspondence and in what context. Who is at the front of his mind when he writes to Britain?

The Pope’s exhortations, for instance, are sometimes ‘to all the faithful of Christ’, But far more often they are directed to the King of England. The Pope is often preoccupied by encouraging the king to ‘incline his mind to peace’, often with France but in one case with ‘him who rules the Kingdom of Scotland’. A phrase which manages to imply the Pope has momentarily forgotten who exactly that is.

Another exhortation urges Prince Edward to ‘use every effort’ to free his father from captivity. This relates to Henry III’s capture by the dastardly ‘democrat’ Simon de Montfort. By the time of the date on the document, Edward had already defeated de Montfort at Evesham so we can see that while the Pope displays a reasonable knowledge of events in England, he’s a little behind the times. Other bulls show the Pope acting in this conflict to excommunicate followers of De Montfort and then sending encouraging notes if they switch sides – he does this with Gilbert de Clair, the Earl of Gloucester. This is just one of many examples of the Pope’s support for the established order. Innocent III gives King John’s rebellious barons just eight days to come to the King’s aid or be excommunicated.

Typing in ‘excommunicate’ shows us those in Britain who enter this most serious of the Pope’s bad books. Poor Robert the Bruce appears to have been excommunicated three times but the Knights Templar also get a number of mentions in this context. Anyone foolish enough to ‘lay violent hands’ on a Templar is to be excommunicated and anyone who seizes their goods or even insults them is placed under ‘anathema’: the anathema is the Pope’s power to send people directly to hell when they die.

The grouping provided by ManyEyes can open up all sorts of avenues for exploring these documents: Why does the Priory of the Holy Trinity in London need so much protection? Can kings really not choose a confessor without the Pope’s permission?

I’ve spent ages playing around with the results. Why don’t you see what you can find?

About the blogger

Joseph Pugh is a member of the Education and Outreach team at The National Archives. He has worked for a number of museums and galleries including the National Portrait Gallery and the Petrie Collection and for BBC History. His research interests include comics in the 1950s, how to win a duel and fainting in the 19th century.

Discovery is an innovative and easily usable way for you to explore our collection. This is a first release, which we will continue to develop, so it will be some time before it replaces the existing Catalogue and DocumentsOnline system.

There is a handy browse facility, by government department, which is designed to be much clearer than the browse facility in the existing Catalogue. By searching a phrase or keyword, a list of results is displayed clearly. You can click through to a particular record where the full description is given along with an immediate option to order a digital or printed copy. Far more information is provided upfront compared with the current search, where it may take several clicks before arriving at a page where your search term is visible.

Another enhancement is the option to refine the search by subject, date and/or collection, enabling you to filter your results with ease. A unique feature of the service is the ability to display search results on a map. Geo-referencing our records is a huge task, and there is a long way to go, but it paves the way for opportunities to use our data in new and varied ways in the future.

Technically the system is very different from our existing Catalogue. The benefit of this new architecture is that it can accommodate the addition of other applications with relative ease and grace; we are planning to add our DocumentsOnline and Image library systems later this year. The current Catalogue is operating at its maximum capacity and is growing at several hundred thousand entries a year, so we need a service that can grow with it.

This tool has been designed with users, for users. We need your feedback on the service as we have plans to develop it and add new features over the next few months. Do try out our new Discovery service and let us know what you think works, what doesn’t work, and what can be improved – your thoughts are extremely valuable to us.

Update – 10/04/2013

Development of Linked Data PRONOM has been on hold for a while, but we will be in a position to restart work on Linked Data PRONOM in the second half of 2013.

Update – 27/10/11

Following feedback from the community about our initial vocabulary publication, we have now released a revised version found that can be found here. 

The vocabulary is now available in RDF and Turtle, and attempts to amend a number of issues that were raised with the initial modelling attempt.  Special thanks for comments go out to Alexander Dutton of Oxford University Computing Services, Mathieu of the LUCERO Project, Bill Roberts at OPF, Lisa Colvin of UDFR, Dave Tarrant at Southampton University and Chris Rusbridge.

We look forward to any feedback you have on the new release and hope that we are going in the right direction of travel.

Update – 25/05/11

A draft vocabulary specification for the linked data version of PRONOM is now available.  The document contains specialised information aimed at the linked data community, and we’re putting the vocabulary on Labs as a means of gathering feedback from those with linked data expertise.  If you aren’t familiar with linked data and want to get a deeper understanding of this area of interest, a useful tutorial can be found at: linkeddatatools.com/semantic-web-basics

Also, a reminder of the location of the prototype PRONOM SPARQL Endpoint and API.

Update – 27/01/11

On his visit to The National Archives in December 2010, Professor Nigel Shadbolt, Professor of Artificial Intelligence in the School of Electronics and Computer Science, University of Southampton, and Transparency and Open Data Adviser to UK Government, discussed the Transparency strategy and the impact of open data in supporting social initiatives and in generating economic growth:

Professor Shadbolt talked about linking data to make it easier to uncover. This is exactly the intention of the linked data PRONOM project. The existing data in PRONOM will be accessible though HTTP URIs, allowing users to view the data and follow links to find out more information about that data. This will make it easier to link to the data in PRONOM, promoting the discoverability and reuse of that data, and providing the means to develop the dataset further.

Update – 17/12/2010

Recently we’ve been busy transforming PRONOM data into RDF, experimenting with putting it into a triplestore, and running Puelia, a linked data API maintained by data.gov.uk and Talis on top of the data.

The API provides access to the data in multiple format representations including a very handy HTML visualisation.  A very early prototype of this work is now available, as well as a SPARQL endpoint.

There is still much to do, such as developing various web services (for example, so that it can interact with DROID), and exploring how we will present data from multiple different sources and express provenance.  While we are a long way from completion, we hope it’ll give you a chance to see how you will be able to use the data from the new version of PRONOM – and to post your comments on how we can build on what we have done to date.

Please remember that, should you receive a message stating page not found, or the site displays a blank page, this is only an early prototype. We are still in the process of modelling the data and making it available, and some of the logic resting behind the linked data API is still to be configured. Just click back in your web browser and continue to browse. Please also be aware that the vocabulary we are using currently is in its draft stages and will change before the project is finally put into production.

Linked data and PRONOM – 6/10/2010

The PRONOM registry contains information about file formats, compression techniques and encoding types. Linked data is about linking up related data on the web, to help expose, share and connect data, information, and knowledge through using URIs and RDF.

Initially we will concentrate on modelling and publishing file format data already stored in PRONOM, using linked data standards. This is the largest core of data within PRONOM, and our first step to transform the data will be to convert existing data to RDF to describe features of each format. The new version of PRONOM will be extensible, so at a later stage we will enhance the data model to improve other areas of information in the database.

Eventually we hope to be able to use linked data to populate PRONOM from other external data sources, transparently showing where the information came from, and in doing so develop a more comprehensive technical registry.

We want the new version of PRONOM to be an open source system with a completely open code base.

We’d like to hear your comments on our plans, or suggestions for improving the PRONOM database, below – your input will inform its development.

The National Archives’ Cabinet Papers site offers free access to the complete minutes and memoranda from every Cabinet meeting from 1917 to 1980.

The cabinet papers keywords application shows major figures and concepts extracted from the complete text of the minutes of those meetings and allows the content to be explored in a completely different way.

We are making this data available, along with a variety of other datasets, to encourage web developers to experiment with new applications, online tools and ways of visualising data.

The application shows the names of members of the Cabinet and the top 25 keywords from all the meetings for the period shown. These have been filtered to remove the most common English words. Each entry links back to the original papers that include it which are available to download.

Keywords are displayed annually in peacetime and monthly in wartime when Cabinet met much more regularly.

The result can offer an insight into the major issues of the day and hint at the relative importance of politicians within the Cabinet. Some terms are predictable, such as ’surrender’ in August 1945, while others refer to half forgotten international crises: what was going on with Iran in 1951? Iraq in 1923? Shanghai in 1927?

We hope you enjoy exploring.

We’ve learned a lot in our first few months on Labs, and I’d like to take this opportunity to thank all of you who took the time to try out the beta applications and tell us what you thought. I know that, at times, we haven’t measured up to your expectations, so thanks very much for your patience and understanding, and all of us on the Labs team hope that you continue to test our refined and new applications and share your invaluable feedback.

Research

Research underpins much of what we do here at The National Archives. As an originator and supporter of research, we maintain an innovative programme to develop and test new ideas for the benefit of government, information and preservation professionals, and are recognised by the Arts and Humanities Research Council as an Independent Research Organisation.

Collaborating with academics is important to us and we welcome applications to partner external research projects. See our research priorities for a list of current research challenges, which we use to inform research partnership decisions and to direct the work of the Research Team.

We currently co-host five PhD students who are undertaking a wide range of studies from historical research to cutting edge technology and resource discovery techniques.  We are looking to appoint another PhD student to work on a hugely exciting project on the classification and categorisation of email messages.  This studentship will be fully funded by the Department of Information Science at Loughborough University and The National Archives.  Please click here for more information and how to apply. Do pass it on to anyone you know who might be interested!

If you’d like regular updates about research at The National Archives please sign up to our free research enewsletter.

We are continuing to improve and refine this service because the most common search our users carry out is for names of people. We want this search function to be as easy to use, but also as comprehensive, as possible.

The newly enhanced search tool allows you to look for names both in our catalogues and also in the census returns and other records which are hosted by some of our external partners. We work with these partners to digitise some of our most popular records, with millions of records now available online. Among the external websites you will be able to search are:

Ancestry.co.uk:

1841 to 1891 census records

First World War Army service and  pension Records

Incoming passenger lists (1878 to 1960), alien arrivals (1826 to 1869) and alien entry books (1794 to 1921)

FindMyPast.co.uk:

1911 census

Royal Hospital Chelsea pension records (1760 to 1913)

Outgoing passenger lists (1890 to 1960)

TheGenealogist.co.uk:

Non-parochial and overseas birth, marriage and death records (1567 to 1969)

GenesReunited.co.uk:

1901 census

Searching the partner sites is free but there is usually a fee for downloading images.

Until this service was launched, you may have had to carry out separate searches on these external partner websites. We hope that this unified person search tool will enable you to explore a wider range of records through our website than was possible before. Let us know what you think.

We do this because government websites contain unique data which would be lost if we did not capture it. Originally we saw websites as a publications medium, but, over the past five years, they have become much more interesting as government has engaged in a dialogue with citizens, and Ministers and others have used social media tools such as blogs to communicate directly with the public.

We also help government departments to ensure that links to documents and other information persist, for example ensuring that users of their websites don’t end up with a ‘Page Not Found’ error message. We’ve achieved this by encouraging them to use redirection technology which takes users to the UK Government Web Archive if the page no longer exists on their live website. Try clicking on this link to see what happens: www.justice.gov.uk/publications/consultation-pandemic-flu.htm

This technology was particularly useful immediately after the General Election when lots of content was removed from government websites. We saw a lot of traffic to the web archive, reaching a peak of 146 million ‘hits’ in July 2010.

What is web archiving?

Web archiving began in December 1996 when Brewster Kahle began his amazing programme to archive the whole of the world wide web (see www.archive.org). Very little of the web survives before that date, though there are a few surviving early sites. The Internet Archive have collected some of these – you can see WebCrawler, The Well and the famous Cambridge coffee pot (a very early web cam) here: web.archive.org/collections/pioneers.html

The National Archives has been archiving government websites since 2003, when we started working with the Internet Archive. We were able to inherit some older sites that they had captured, so in our collection we have websites going back as far as 1997. There appear to be no government websites before that date – the earliest Public Record Office (PRO) site is dated from December 1998. This is not really surprising since the early sites were created by sending floppy disks to the government telecommunications agency in Norwich. They could only be updated once a month.

Since 2008, we’ve really expanded our programme to create a comprehensive archive of government’s web presence. The European Archive have been contracted to crawl and host the collection since 2005.

Some facts about the web archive:

• It is free to use and is accessible online: www.nationalarchives.gov.uk/webarchive
• It contains approximately 1 billion documents
• There is a great variety of content, archived over many years
• It has a wide variety of users (government itself, researchers, journalists, the general public)
• Approximately 2,000 websites have been catalogued so far

Read more about web archiving.

Watch UK Government Web Archive: A retrospective, a video illustrating some of the changes that government websites have undergone.

Archiving social media

The Web Archiving team at The National Archives is also exploring possible solutions for archiving social media platforms used by government departments. Using The European Archive we are able to take snapshots of Twitter and Flickr and for example, did a sweep before the general election in May. Many government department Twitter feeds are now crawled on a similar basis to their websites. We are continually looking to improve on the processes used and are exploring more reliable methods and different technologies. We’re happy to share our knowledge and what we’ve learnt so far and would also be interested to hear from anyone developing similar solutions. Please email webarchive@nationalarchives.gsi.gov.uk.

I’d like to apologise for any issues you may have experienced when visiting our Labs site recently. The level of interest has been far greater than we expected, and our Labs site couldn’t quite meet the demand.

All the prototypes on the Labs site (including the Domesday tool) are trials which have been produced at very low cost, using new technology, new ideas and new ways of processing data. They are not perfect and need to be built on.

Before we invest, we want to know what you think about them and whether you would find them useful. This will help us focus our limited funding and resources in a way that supports you, our customers, best.

So far this has been a success and given us a strong steer on what works, what doesn’t and what you would like to see. It is great to see so much interest and shows we are going down the right path in our developments. The level of interest in the site yesterday (particularly the Domesday tool) caused a number of technical issues, which we are now working to resolve. We will use the lessons learnt this week to improve both the robustness of the Labs site itself and how we tell our customers about it.

In spite of these teething problems, we have been able to get very useful feedback on the automatic geographic referencing we trialled with the Domesday tool. We used automated software, rather than a manual process, to geo-reference this data. We’ve now learnt that this works on the whole but that there are a few specific issues. We are now comparing the computer-generated data with the most up-to-date scholarly list of Domesday names to help resolve any issues.

I hope that you will continue to take an interest in our online developments and tell us what you think of them. We are very excited at being able to open up our collection in new and innovative ways, and we very much value your support in doing so.

Using the interactive map

A current OpenStreetMap map shows all of the places which are mentioned in Domesday. Clicking on a place brings up the place’s name in 1086, so you will be able to find out your home town’s eleventh century name. You can discover the names of the people who held land there and, for a charge of £2.00, you can download a copy of the relevant page along with an English translation.

We’ve added a postcode search too. So, inputting TW9 4DU will take you to a map of Kew and the surrounding area; if you click on Petersham, you will learn that in 1086 there was a fishery, which captured 1,000 eels and 1,000 lampreys every year.

Why it matters

Did you know that there were about 46 vineyards mentioned in Domesday Book? Most seem to have been planted since the Norman Conquest of 1066 and all were south of a line between Ely and Gloucestershire. Clearly these were a reflection of the Norman fashion for drinking wine and probably evidence of a warmer climate than existed a few centuries later. This kind of additional geographical and environmental information is one of the most significant aspects of Domesday.

As an example of the kinds of information you can find in Domesday, we’ve added special features to the maps so you can see the number of places with Viking names, reflecting the Danish settlement after 865CE, and also the names of places where cranes (once a common English bird) can be found.

Domesday tells us great deal about the relationship between the King, the church, the great landlords and ordinary people. Something of the flavour of life in Old Romney can be learned from this entry:

The same Robert has 50 burgesses in the borough of OLD ROMNEY; and from these the king has every service, and they are by reason of service at sea quit of every customary due except these 3: theft, breach of the peace, and highway robbery.

How you can help

The links between the maps and Domesday were generated automatically so there may be a few errors. Please let us know if you find any. No prizes – just our grateful thanks.

This tool enables you to search and browse over 90% of the places mentioned in Domesday Book.

Here at The National Archives, we understand the importance of involving you, our users, in the development of new services to make sure that they are as usable and useful as possible. At every stage of the design process we take account of the needs, wants and concerns of the people who will be using the new services. We have colleagues at The National Archives dedicated to improving customer experience, who gather feedback from our users via various channels – online and in person. This helps us to make sure we are delivering the right services to our users.

The idea behind Labs is to allow you to see prototypes and early designs and leave your feedback. We take your feedback – both good and bad – and use it to shape the design and development of the projects that we are working on. For instance, based on what you told us, we are developing a postcode finder for our geo-referencing tools, such as the UK history photo finder.

It doesn’t just end at Labs though. We also conduct user testing sessions at various stages of the design process. Here, we invite a small number of people who represent typical users of our services to test out our designs and ideas. These sessions are run with individuals rather than in groups, and usually last around an hour. We ask the users to carry out a number of tasks and realistic scenarios, and to tell us what works, what doesn’t work, what they liked and what they didn’t like. This then allows us to refine our designs and ideas. This could be anything from how we’ve labelled a button to the position of a feature on a screen.

We will also be holding focus groups at various stages of the Rediscovering the Record project for you to give us further valuable feedback on what we are doing and where we need to go. Focus groups allow our users to discuss with each other, as well as with us, their thoughts on project developments. Our next focus group will be held at on Tuesday 12 October at The National Archives in Kew. If you’d like to take part to share your views, please email marketing@nationalarchives.gsi.gov.uk with ‘Focus group’ in the subject line.

Finally, we conduct creative workshops with front-line staff to get their feedback on how these developments will affect the visitors to our reading rooms, who they deal with on a daily basis. By involving those who deal regularly with The National Archives’ visitors, we feel we can provide the best possible customer service.

All of this ensures that the final product is shaped on your needs and requirements, so we can provide access to our services in the most user friendly way.

This week, I am going to talk about the challenges we are experiencing with map-based applications, and also say something about the work we’ve been doing on identifying places in our records – a process called geo-referencing

Mapping

We know that some our early efforts have not been as usable as we would hope. Obviously, the issues with IE8 are disappointing, to us as well as to our users, and we are working on solutions and alternatives. While we can’t make everything on our beta site meet our website standards of usability and accessibility, we want as many people to try out these applications as possible and to know what they think. That’s why we’ve recommended that users experiencing problems access applications in IE8 compatibility mode, rather than us take things offline.

Please let us know if we could have communicated this more effectively. Are we answering your questions? And are you seeing questions that other users have asked and been answered further up the comment thread?

This is The National Archives’ first interactive commenting feature and is a learning process for us. Tell us what you want from it.

Geo-referencing

Our records contain millions of references to places in the UK and overseas. We want to make it easy to find these and, ideally, to plot them on a map so that you can, for example, look at a map of England, see which places are mentioned in Domesday Book, and then click through to the record.

While it is easy for anyone reading a document to recognise that ‘house in Willesden’ or ‘fields round Abergavenny’ relate to places, it is much harder for computers to figure this out. We first need to be able to identify place names in records and then we need to look them up in a gazetteer to identify their precise locations. The process of identifying place names is called geo-parsing and the process of identifying their locations is called geo-coding.

This is not as easy as it seems, because:

• Geo-parsing systems often rely on contextual information such as the word ‘near’ before the location or ‘city’ after the location, to identify a possible location. However our digital archival data sometimes contains very little contextual information.
• When it comes to geo-coding, the big problem is that there are many places with the same name (there are 14 Newports in the UK) and in many cases words are used both as place names and personal names (for example the town Wellington and the Duke of Wellington). We are using a number of techniques which allow us to use contextual and historical information to determine whether a word is a personal name or a place name and, if it is a place name, which precise place is being referred to.
• Often, place names used to be spelled differently. For example, the modern place of Loughton, Essex was spelled Lochintuna in Domesday Book, a name that naturally does not appear in modern gazetteers. However, using historical and archival information about place names, we are able to learn spelling patterns which enable us to identify the modern spelling of old place names.

The first applications that build upon our automated geo-referencing will become available here soon – we are planning to launch a map-based search of Domesday Book. As with any large-scale automated process of this kind, there will be a number of mis-identified and mis-located places. However, we are developing a series of methods for identifying such errors and allowing users to correct them.

I will be talking about other site features and new releases in later blog posts. If there is anything you want to know more about, do let me know in the comments section.

An automatic categorising tool allows individual document descriptions to be tagged using a list of subject categories (see below). Subject categorisation is no longer limited to collection level descriptions but now extends across metadata for files, boxes and bundles held by The National Archives. This has been achieved by building sophisticated boolean queries identifying which records should be tagged under different categories.

We have now exported the taxonomy and plugged it into a user interface, allowing testing and tuning to commence. So far, we have created over 10 million tags across The National Archives Catalogue data. Multiple tagging of one record is allowed within our framework, so we will focus on the breadth and depth of tagging during the testing process.

We welcome your feedback on these developments.

As part of Rediscovering the Record we are planning to improve the ways in which you can search. The National Archives can search for about 30 million records. These include the 11 million descriptions of documents we hold at Kew and records held locally by other archives, which are described in the National Register of Archives. We know that it’s tricky to find what you need in such a large volume of rich and diverse material.

We’ve looked at whether we can adopt a more Google-style approach to searching the archives, but have come to the conclusion that we can’t. Major search engines rank the relevance of a website using complex algorithms. One factor in this is a measure of how many other websites link to it. This approach doesn’t translate well to our archival records.

Instead, we are doing two things.

First, we are building a subject classification to group all records into topics. We think this will help you filter your results based on the intent of your search. For example if you type in Wellington, it will sort the results into different categories depending on whether they relate to: the hero of Waterloo, or a pub named after him; the bomber; the place in Somerset; or the boot. The topics vary from suffragettes to sewerage, from food and drink to farming. Have a look at our initial topic list – it will be available on Labs soon – and let us know what you think. We hope it will help you find the pearls among the seaweed much faster!

Secondly, we are producing a list of prominent historical people – essentially everyone we can identify from standard reference books, so that if you type in Churchill, it will distinguish between the hero of Blenheim, the Prime Minister and the journalist. So far we have about 40,000 people in this list and plan to expand it further.

We also plan, in time, to develop more accurate lists of historical place names to resolve complications around some descriptions, for example to account for the spelling of place names that have evolved or changed completely (starting with an online index to Domesday Book).

If you have any comments or questions, please use the Comment facility below.

This is my first blog on our Rediscovering the Record project. I want to first of all describe the overall thinking behind the project, and in the next few months I will look in more detail at some of the improvements we will be making.

The project will create new, improved and innovative ways of accessing our records and ensure they are easily accessible in the future. The project has four major parts – a new catalogue, a new search tool, geographical-based searching and user involvement.

The National Archives currently hosts two major online catalogues – The National Register of Archives (launched online in 1995) and the Catalogue of The National Archives’ own holdings (launched in 1998). Both of these have grown rapidly – The National Archives Catalogue has doubled from 5 million entries in 1998 to over 11 million today. This rate of growth is unsustainable on our current technology platforms so we plan to develop a new, more robust catalogue and to bring the two together into one system, with a single interface.

As part of the redevelopment of the catalogues we will build a new and improved search tool, to provide more detailed, helpful subject information and make searching for names and places much easier.

We are also developing map-based searching – making it possible to click from a map to a record. We have already launched some pilot applications (UK history photo finder and Valuation Office map finder) on Labs and we are planning more for later in the year.

Finally, we intend to make it possible for users to get involved in our cataloguing work. In future, you will be able to add descriptive information to our catalogues. Equally importantly, developers will be able to download data from our catalogues to reuse as they think fit. This could be used in smartphone apps or mashed up with other catalogues or historical data. What can be done is only limited by your imagination.

You have already provided invaluable feedback through your comments on the site, telling us what you think of the concepts and how the technology works (or could work better!) for you. All of your contributions are read and shared around the Labs team – please keep talking to us.

We intend to have much of the groundwork finished by the end of March 2011 with further developments going live in the coming months.

In my next post I’ll be talking about our shiny new search project.

Get started with the Valuation Office map finder.

We have detected your browser does not have Javascript enabled. You must have Javascript enabled to use this application.

The maps are Ordnance Survey maps covering the whole of England and Wales, but please bear in mind that records do not survive for all places.

The maps serve as the means of reference to more than 95,000 Valuation Office Field Books which contain descriptions of more than 9 million individual houses, farms and other properties, detailing the use and value of lands and buildings, and naming their owners and occupiers.
Please note that Valuation Office Field Books can only be viewed at The National Archives.

For more information on the Valuation Office Survey, its history, contents and how to access its information see the research guide Valuation Office Records: the Finance (1909-1910) Act.

Please let us know what you think by leaving your comments below. We’d like you to tell us:

• Do you find this tool useful?
• How would you use it for your research?
• What other TNA data would you like to see used in this way?
• How else would you use this technology?
• How do you think it could be improved?