Technical specifications

This page provides technical information for building and maintaining your collection in Trove. It describes transfer protocols, record formats, and how we use NUC symbols to uniquely identify your collection.

Please have a read through and contact Partner Support with any questions. You may also want to review this list of systems we are already working with.

For information on creating great records - read part one and part two of our two-part Trove blog series.

Transfer Protocols

Trove can work with a variety of transfer protocols to harvest records from contributors’ systems. These include Sitemaps, the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) and custom APIs.

While we can work with other setups, if you are building a new system with the intent of working with Trove, we recommend building a Sitemap. It has many benefits outside of Trove, making your website more discoverable to search engines.

In the past we have also worked with Harvest Control Lists and RSS feeds. We no longer recommend these for new collections. Instead, consider using a Sitemap or OAI-PMH.

Sitemap

A Sitemap is a standard XML protocol. It lists all the web pages within a website. Search services like Google, Yahoo and Bing use a Sitemap as a guide around a website, to help them find and visit every page. They use this data later to point users towards the page where their search term exists.

Trove uses Sitemaps in a similar way, as a list that points to the different web pages where records are available. Every URL in the list is visited, and Trove will attempt to ingest each page as a record. Pages not mentioned in the Sitemap are ignored.

Many popular web hosting platforms such as WordPress and Drupal include Sitemaps functionality by default, or as an optional plugin.

To work with Trove’s web crawler a website needs to have three layers of files:

  • robots.txt - A plain text file that instructs webcrawlers in how they should move around your website. It can contain an optional line 'Sitemap: ' followed by the URL of a Sitemap file, for example Sitemap: http://www.sitemaphost.com/sitemap-host1.xml To contribute to Trove you will need to include this line.
  • Sitemap or Sitemap Index file – a list of URLs. These resolve either to more Sitemap files, or to web pages that can be treated by Trove as single record pages.
  • Single record pages – a single HTML or XML file that will represent a single record in Trove. This might be a single episode of a podcast, a single artwork, a single journal article. We recommend that this be the same page any human visitor would see, with some extra hidden data underneath for services like Trove, as well as the big search engines. This extra data is hidden in <meta> tags, see the HTML record section for more information.

Trove can read both Sitemap and Sitemap index files. The ‘lastmod’, ‘changefreq’, and ‘priority’ elements are currently ignored.

Sitemap websites can be crawled on a daily, weekly, monthly, bi-monthly, quarterly, half-yearly or yearly schedule. Nominate the frequency that suits you, considering how often your content is added or changed.

Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH)

OAI-PMH is a web-based protocol that defines a standard way to move batches of records between systems. An OAI http request is sent from the service provider, Trove, to a data provider, a repository. A batch of 100 records is sent back in XML. A URL to access the next batch of records (called a resumption token) should be available after the last record.

As each batch is received the records are immediately loaded into Trove. The following week an update harvest is conducted, and only records added, updated or deleted in the intervening time are returned. Trove refreshes accordingly.

A number of popular library systems, institutional repositories, archival management platforms and other record management systems have in-built OAI-PMH functionality. For others, an optional plugin may exist. This means you may already have the technical capability to contributing records to Trove via OAI-PMH. 

The standard for OAI-PMH can be found at http://www.openarchives.org/pmh/

To interact with a working OAI-PMH repository see the Library of Congress Sample OAI-PMH requests

A good overview of the OAI-PMH process is available at the CAIRSS website.

Custom API

Trove can be configured to work with your organisation’s custom API.  We’re best placed to work with APIs that

  • Are RESTful.
  • Offer XML or JSON output
  • Output Dublin Core or similar standard metadata
  • Include notifications of records that have been deleted
  • Can limit the number of records returned per request to less than 100, and can paginate through results
  • Can provide incremental harvests (“show me all records updated  since 2014-06-18")
  • Provide all metadata fields in the search results, without a second call for the full individual record

None of these are hard and fast rules, they do make setup and maintenance quicker at the Trove end and minimise the number of calls to your API.

To integrate with any custom API requires specialised work by the Trove team. The process of interoperating with a custom API can therefore take longer to setup than other standardised harvesting methods. Please contact Partner Support to discuss how Trove can work with your API.

Record formats

Trove specifies the standard data elements it expects from all contributors’ collections. This is called a schema. While the transfer protocols provide a way to move the batches of records into Trove, the schema specifies what kinds of information Trove should expect for each record.

Items do not have to be fully available online for free (e.g. a digitised picture or a full thesis). It may be a link to a finding aid describing the box an item is stored in, or an abstract for an article.

For a full listing of fields included in Trove's metadata schema, see the Data Dictionary.

Published and unpublished material

Trove currently works with a variety of metadata schemas. The majority of published and unpublished material is described using Dublin Core.  Some DC Terms and NISO Access and Licence Indicators (NISO RP-22-2015) fields are also accepted.

Each record contributed to Trove must include these three mandatory fields:

  • Identifier (including a linktype)
  • title
  • type

It is highly recommended that these six fields are included:

  • contributor
  • creator
  • date
  • description
  • subject
  • rights
  • free_to_read
  • license_ref

If known, it is best practice to also include the following:

  • coverage
  • format
  • language
  • publisher
  • relation
  • source

Records should be presented in XML, or HTML with meta tags to store the separate fields. Your records will be aggregated into Trove much faster if your ‘identifier’ tag contains specific labelled information. For example:

<dc:identifier>http://nla.gov.au/nla.pic-an9186335-t</dc:identifier> does not let us know what the link actually contains, and time is spent to work out if it is the actual content, a thumbnail, or something else.

<dc:identifier linktype="thumbnail">http://nla.gov.au/nla.pic-an9186335-t</dc:identifier> lets us know. Read more in the data dictionary.

MARC standard records are accepted via the service Libraries Australia.

People and organisations

Records describing people and organisations are stored within Trove in Encoded Archival Context - Corporate Bodies, Persons, and Families (EAC-CPF) format.

Each record contributed to Trove must include these five mandatory fields:

  • agencyCode (ISIL or NUC Symbol)
  • fromDate
  • nameEntry (forename)
  • toDate
  • recordId

It is highly recommended that these four fields are included:

  • agencyName
  • entityId
  • localDescription (nationality)
  • nameEntry (surname)

If known, it is best practice to also include the following:

  • biogHist
  • cpfRelation
  • identifier (thumbnail)
  • localDescription (gender)
  • occupation
  • otherRecordId

These records were previously accepted as part of the People Australia Project.

Example: XML record

An XML record can contain both Dublin Core and Picture Australia elements for images, or the extended form of Dublin Core that Trove implements. This record describes a picture entitled Captain Cook, from an original picture by Dance in the gallery of Greenwich Hospital.

<?xml version="1.0" encoding="UTF-8" ?>
<record><oai_dc:dc xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://purl.org/dc/elements/1.1/ http://www.openarchives.org/OAI/2.0/oai_dc.xsd">
< dc:title>Captain Cook, from an original picture by Dance in the gallery of Greenwich Hospital [picture] /</dc:title>
< dc:creator>Scriven, Edward, 1775-1841.</dc:creator>
< dc:contributor>Dance, Nathaniel, 1735-1811.</dc:contributor>
< dc:contributor>Society for the Diffusion of Useful Knowledge (Great Britain)</dc:contributor>
< dc:coverage>1840 - 1849</dc:coverage>
< dc:description>"Under the superintendance of the Society for the Diffusion of Useful Knowledge."; Beddie, 3367.; Also available in an electronic version via the Internet at: http://nla.gov.au/nla.pic-an9186335; S3582.</dc:description>
< dc:format>1 print : engraving ; plate mark 28 x 20 cm.</dc:format>
< dc:identifier>nla.pic-an9186335</dc:identifier>
< dc:rights>You may save or print this image for research and study. If you wish to use it for any other purposes, you must contact the <a href="http://www.nla.gov.au/digicoll/copyright.html">National Library of Australia</a> to request permission.</dc:rights>
< dc:publisher>London : Published by Charles Knight, [184-?]</dc:publisher>
< dc:subject>Cook, James, 1728-1779 -- Portraits.</dc:subject>
< dc:subject>Explorers -- Great Britain -- Portraits.</dc:subject>
< dc:type>Image</dc:type>
< dc:identifier linktype="thumbnail">http://nla.gov.au/nla.pic-an9186335-t</dc:identifier>
< dc:identifier linktype="fulltext">http://nla.gov.au/nla.pic-an9186335</dc:identifier>
< /oai_dc:dc>

Records for items without thumbnails do not require the extra Picture Australia elements, however it is helpful to include the linktype attribute. This indicates to Trove if the item is fully online. For example this record for a digitised text contains only Dublin Core elements.

<?xml version="1.0" encoding="UTF-8" ?>
< record xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ http://www.openarchives.org/OAI/2.0/oai_dc.xsd">
< dc:title>Some thoughts on examinations [electronic resource]/</dc:title>
< dc:creator>Examiner.</dc:creator>
< dc:publisher>[Oxford] : Privately printed, 1906.</dc:publisher>
< dc:format>11 p. ; 22cm.</dc:format>
< dc:coverage>1906</dc:coverage>
< dc:date>1906</dc:date>
< dc:description>Electronic reproduction. Canberra, A.C.T. National Library of Australia,2010.</dc:description>
< dc:contributor>Examiner.</dc:contributor>
< dc:type>Text</dc:type>
< dc:identifier linktype="fulltext">http://nla.gov.au/nla.gen-vn4807078</dc:identifier>
< dc:identifier>nla.gen-vn4807078</dc:identifier>
< dc:source>Item held by National Library of Australia</dc:source>
< dc:rights>You may save or print this image for research and study. If you wish to use it for any other purposes, you must contact the National Library of Australia to request permission.</dc:rights>
< /record>

Example: HTML record

A record in HTML needs to include the meta element within the head section of the file. Other information may be contained in the body, allowing meta tags to be embedded in existing pages. This record describes a picture entitled Captain Cook, from an original picture by Dance in the gallery of Greenwich Hospital.

<html>
< head profile="http://dublincore.org/documents/dc-html/">
< link rel="schema.DC" href="http://purl.org/dc/elements/1.1/" />
< link rel="schema.PA" href="http://www.pictureaustralia.org/schemas/pa/" />
< meta name="DC.title" content="Captain Cook, from an original picture by Dance in the gallery of Greenwich Hospital [picture] /" />
< meta name="DC.creator" content="Scriven, Edward, 1775-1841." />
< meta name="DC.contributor" content="Dance, Nathaniel, 1735-1811." />
< meta name="DC.contributor" content="Society for the Diffusion of Useful Knowledge (Great Britain)" />
< meta name="DC.coverage" content="1840 - 1849" />
< meta name="DC.description" content="&quot;Under the superintendance of the Society for the Diffusion of Useful Knowledge.&quot;; Beddie, 3367.; Also available in an electronic version via the Internet at: http://nla.gov.au/nla.pic-an9186335; S3582." />
< meta name="DC.format" content="1 print : engraving ; plate mark 28 x 20 cm." />
< meta name="DC.identifier" content="nla.pic-an9186335" />
< meta name="DC.rights" content="You may save or print this image for research and study. If you wish to use it for any other purposes, you must contact the National Library of Australia to request permission." />
< meta name="DC.publisher" content="London : Published by Charles Knight, [184-?]" />
< meta name="DC.subject" content="Cook, James, 1728-1779 -- Portraits."
< meta name="DC.subject" content="Explorers -- Great Britain -- Portraits."
< meta name="DC.type" content="Image" />
< meta name="PA.thumbnail" content="http://nla.gov.au/nla.pic-an9186335-t" />
< meta name="PA.viewcopy" content="http://nla.gov.au/nla.pic-an9186335-v" />
< meta name="PA.location" content="http://nla.gov.au/nla.pic-an9186335" />
< /head>
< body>
< /body>
< /html

Using a NUC Symbol to identify your collection

Records in Trove are linked to a National Union Catalogue (NUC) Symbol. This is a brief collection identifier made up of 2-10 letters. NUC Symbols are issued by the National Library of Australia and maintained in the InterLibrary Resource Sharing Directory. Every NUC Symbol is unique and allows Trove to identify the organisation holding the item. It can also identify a sub-collection. A NUC Symbol is added to each incoming record by the Trove harvester.

Organisations are welcome to nominate their pre-existing NUC Symbol or request a new one for a particular collection. They can be obtained by filling in the New Service Notification Form at the ILRS Directory, or the Trove team will set one up on your behalf when you contribute to a new collection.