Web archiving at the Borthwick

We will always seek your consent and written permission before crawling your website as part of our web archive program. At this time, we will also discuss what web archiving involves, and share information with you about how frequently your site will be captured and how archived copies will be accessed.

University of York websites are captured as part of our University Archive.

The frequency with which we crawl your website will depend on collection guidelines and the nature of individual websites. Websites that are actively maintained and updated will likely be captured and recaptured at regularly scheduled intervals, such as semi-annual or quarterly.

In some cases, a website may only be crawled once or for a specified time period. Occasionally, we may choose to discontinue regularly scheduled crawls. These cases might occur if:

• The value of the site is limited to a specific time period (such as a conference, one-time event, or the work of a temporary committee);
• The site has reached its end of life and will be taken down after harvesting;
• The targeted website has exhibited no or minimal change for three consecutive years (as verified by Archive-It curatorial tools and manual review by staff). 

We crawl websites at a rate designed so as not to interfere with performance. For actively updated websites, crawls will generally be run quarterly or semi-annually, and last for a few days. Very large websites may need to be crawled over a period of weeks. Once a crawl is complete, the crawler no longer interacts with your server. It is unlikely that Archive-It’s crawlers will impact the performance of your website, but if you encounter any issues or have any additional questions, please contact us at borthwick-institute@york.ac.uk. 

No. The Borthwick does not archive password-protected content. If you believe that your site’s password-protected content should be included as part of our web archive, please contact us.

In order to create an accurate snapshot of your website for future researchers, we aim to capture as much of your site as possible. We will not bypass robots.txt exclusions by default, but may contact you to request changes to rules or seek permission to override them where such exclusions prevent the capture of content. 

No. By archiving your site, the Borthwick is preserving a static snapshot of your site at a particular time. The hosting, management and maintenance of your live website remains your responsibility.

Yes, downloadable media, audio and video files can usually be captured, although content hosted on third-party services like YouTube, Vimeo or Soundcloud can sometimes be challenging. For this reason, we may ask you to provide separate copies of media, audio and video files where possible, which we will store outside of the web archive and which may provide a more robust solution for long-term access.

Archived copies of sites won’t render files that are not linked and have to be retrieved from a database via user queries (for example, when a user must execute a search in order to retrieve and access a file). 

The Borthwick makes sites archived as part of its web archiving program freely available to the public via our Archive-It partner page, where website level metadata is supplied to allow for browsing and full-text search. A prominent banner informs users that they are viewing an archived web page, as opposed to the live site. Relevant links to archived copies of sites will also be included in collection guides and/or catalogue records.

By default, archived sites undergo a six-month embargo period before being made available. Where appropriate and in discussion with site owners, selected content may be subject to longer embargo periods.

If your website includes content that is challenging for Archive-It’s crawlers, Borthwick staff will work with you to find alternative solutions for capturing that content. This may mean storing copies of challenging files separately to the web archive and providing access via alternative means.

However, owners wishing to optimise their site design to support more complete archiving may be interested in Columbia University Libraries’ Guidelines for Preservable Websites.

Users may access the collection for non-commercial research purposes and private study. Content within the collection may be subject to intellectual property rights governed by local, national and/or international laws or regulations. In using this content, you are responsible for abiding by all applicable laws in connection with your research.

Users are also responsible for ensuring that they use archived web resources in an ethical manner. Ethical use includes ensuring that archived web content is not represented as the live or most current version of a site, as well as accurately citing any archived web content used in research.

In addition, users are responsible for any personal data concerning living individuals obtained from archived web resources. When you capture or take away personal information you become the data controller of this information, and are liable for it and any subsequent use made of it. It is your responsibility under Data Protection Law to ensure that your use of any personal data accessed via the Borthwick Web Archive is fair and lawful.

There are a number of reasons why the archived copy of a website may appear to be incomplete:

• Some types of content are challenging or impossible to capture and/or reproduce, including JavaScript-driven navigation menus, streaming audio and video, and dynamic form and database-driven content.
• We only capture publicly accessible content, so restricted or password-protected site pages will not be collected.
• Typically, the scope of our crawls does not include links to content hosted on other websites. If you find that a link on an archived page no longer works, double check the URL–you will likely find that it belongs to a different site.