Web Curator Tool

The Web Curator Tool is a tool for acquiring web material, such as websites, web pages, and other documents you might find on the internet. The collected web material can then be stored and preserved in a digital archive.

The National Library uses the Web Curator Tool to collect New Zealand's online heritage. The tool has been developed as an open-source project for the benefit of other organisations building collections of web material.

Download the Web Curator Tool

The Web Curator Tool, along with manuals, FAQ documents, source code, development documentation and further information, is available on the Web Curator Tool website.

Web Curator Tool website

Selective harvesting and domain harvesting

Web harvesting is the process of downloading web material. There are two basic approaches: selective harvesting, and domain harvesting.

In selective harvesting, high-value web material is identified, and then scoped for harvest. A domain harvest involves attempting to harvest all the web material within an internet domain; for example, all the websites whose URLs end in '.nz'.

The Web Curator Tool supports selective harvesting by allowing curators, librarians or archivists to perform several crucial tasks:

• selecting the web material to harvest
• setting a schedule for harvesting the web material
• performing the harvest, by downloading the scoped web material at the scheduled time
• reviewing the harvest results to ensure the web material has been downloaded properly
• sending the web material to a digital archive for storage.

Web archiving community

The International Internet Preservation Consortium (IIPC) provides a mailing list for web curators and other practitioners interested in sharing information and experiences about web harvesting.

Web curators mailing list – IIPC website

The Web Curator Tool development project

The Web Curator Tool was jointly developed by the National Library of New Zealand and the British Library. We collaborated with the National Library of Australia, the UK Web Archiving Consortium, and the International Internet Preservation Consortium. Sytec Resources Ltd. was selected to design and build the Web Curator Tool software in a competitive tender process.

The first version of the tool was completed and released as open-source software in September 2006. There have been several follow-up releases, the most recent in August 2007.