Web archiving expertise
I help libraries and collecting institutions build, operate and improve web-archiving systems.
I develop and maintain widely used open-source software for web crawling, indexing and replay, and have extensive experience operating these systems in production at national scale.
Services
Web-archiving systems
Architecture, implementation and modernisation of web-archiving systems, from individual components to complete production environments. Customisation and integration of open-source web-archiving tools.
Web archive health check
A focused technical review of an existing web-archiving system, covering capture quality, crawler configuration, indexing and replay, infrastructure and operational risks. The result is a practical set of findings and prioritised recommendations.
Replay and capture troubleshooting
Investigation of capture, indexing and replay problems, including issues that cross several parts of the system.
Training and configuration
Technical training and practical assistance with configuring, deploying and operating web-archiving software.
Need technical help with web archiving? Get in touch.
Experience
I spent 20 years at the National Library of Australia working across software development and systems administration, with hands-on responsibility for the architecture, performance and reliability of large production systems.
My work included end-to-end technical responsibility for the Australian Web Archive, its 1.3 PB collection, and the systems supporting crawling, WARC processing, indexing, replay and preservation. I also helped design, deploy and operate the Solr search cluster and web infrastructure behind the Trove discovery service and the Library's preservation storage system.
Selected work
- Heritrix
- I am the lead maintainer of Heritrix, the web crawler used by many national libraries and archives. My work has included modernising the codebase and implementing HTTP/2 and browser-based crawling support.
- OutbackCDX
- I wrote OutbackCDX, a high-performance CDX index server for web archives containing billions of records. Unlike traditional sorted text file indexes, it can be updated incrementally without rebuilding the entire index and uses roughly a quarter of the storage. It is used by major collecting institutions around the world, including the Library of Congress, the British Library and the Internet Archive.
- PANDAS 4
- I built PANDAS 4, a collaborative web-archiving workflow system used by the National Library of Australia and participating state libraries to coordinate curatorial selection, collection building, crawl scheduling and quality assurance across institutions.
- jwarc
- I wrote jwarc, a high-performance Java library and command-line tool for processing WARC files. Common Crawl features it in their Java data tutorial, and the Royal Danish Library built a large-scale CDX indexing workflow around it.
- pywb
- I co-maintain pywb and contribute fixes and improvements across the wider Webrecorder replay stack, including wombat.js and wabac.js, browser-side libraries that help modern archived sites function correctly.
- WARC specifications
- I created the community-annotated WARC 1.1 specification. I also propose extensions to the format, including the WARC-Protocol field for recording the network protocols used during capture.
Contact
If you are planning a web archive, modernising an existing system or troubleshooting capture and replay problems, email hello@moaduda.com. A short description of your system, challenge and timeframe is helpful.
You can also find my work on GitHub.
I also take on software development and systems consulting projects outside web archiving.