Whole of domain web harvest
Learn about the work we do to collect and preserve websites and the New Zealand web domain harvest that takes a ‘snapshot’ of the .nz domain.
Why does the National Library collect websites?
The National Library preserves New Zealand's social and cultural history, in the form of books, newspapers, photographs, websites, blogs and videos.
The New Zealand Web Domain Harvest recognises the importance of the internet in all areas of New Zealand society and culture by taking a ‘snapshot’ of the whole .nz domain as it exists on the web during the time of harvesting. The Library’s first domain harvest took place in 2008. Further harvests were run in 2010 and 2013. They have been run annually since 2015.
The National Librarian is authorised to harvest websites by the National Library of New Zealand (Te Puna Mātauranga o Aotearoa) Act 2003 and the Minister’s National Library Requirement (Electronic Documents) Notice 2006.
National Library of New Zealand (Te Puna Mātauranga o Aotearoa) Act 2003
Minister’s National Library Requirement (Electronic Documents) Notice 2006
What is a domain harvest?
The Library undertakes two streams of web archiving: selective harvesting and domain harvesting.
Selective archiving is where Library staff select high-value websites for inclusion in our collections. The Library has been selectively harvesting since 1999.
Domain harvesting is an attempt to harvest as much material as is technically possible with a minimum of human intervention. It is called "domain harvesting" because the simplest approach is to try to harvest an internet domain, such as the NZ (or ".nz") domain for New Zealand.
Technical details
The technical parameters of the harvest were developed after consultation with the public and internet stakeholder groups. We acquire:
websites that fall under the .nz and .kiwi top-level domain codes
websites that fall under .com, .net and .org that can be programmatically determined to be hosted on machines that are physically located in New Zealand
selected websites based overseas that are covered by the provisions of the National Library of New Zealand Act (2003)
The Library commissioned the Internet Archive (an American-based not-for-profit) to perform the harvest on our behalf.
The crawler uses the user agent string NLNZ_IAHarvester[year].
Some web harvesting statistics
Year | Number of URLs harvested | Size of the harvest (compressed) |
|---|---|---|
2008 | 138 million | 2 TB |
2010 | 140 million | 6 TB |
2013 | 220 million | 11 TB |
2015 | 231 million | 10 TB |
2016 | 311 million | 16 TB |
2017 | 638 million | 24 TB |
2018 | 671 million | 33 TB |
2019 | 244 million | 15.5 TB |
2020 | 353 million | 19.3 TB |
2021 | 659 million | 27.4 TB |
2022 | 368 million | 29.8 TB |
2023 | 503 million | 37.59 TB |
Contact us
If you would like to send us a question, or comment on the domain harvest, email Web.Archive@dia.govt.nz.