CollectionBuilder's Scalability?

[hwlong0305]

Hello. I heard about CollectionBuilder at Digital Library Federation Forum in November and am interested in using it for one of our large photo collections. We currently have approximately 25,000 records/files and anticipate the collection will slowly grow to 800,000 records/files. Can anyone in this community speak to CollectionBuilder's scalability? What are its upper limits? I assume the CSV template is the best to use for this use case... Is that correct? Thanks in advance for any advice you can offer!

[evanwill]

@hwlong0305 wow, that sounds like a huge collection! I believe that will become too cumbersome to maintain as a single collectionbuilder project and would likely exceed good user experience at that size...

A static site's efficiency is ultimately limited by the size of data files it needs to load to render the content. In collectionbuilder, your metadata is parsed down into the data necessary for each page (for example, the Browse page data has only the fields that will be displayed or used in the browse card visualizations) to minimize the load time--but at that scale, the file size is going to get big... just how big depends on how much text is in your metadata.

Our largest collection is the Argonaut Archive, which contains 6,500 issues of a newspaper. The metadata does not have much text in it. You can browse around the site to get a sense of how long it takes to load the pages, etc. Another large example of just a table visualization is our legacy site HistPhoto which loads 92,000 items--the table stays responsive, however the json file it needs to load is 10MB, which isn't ideal for most users.

With 25,000+ items, the Timeline, Browse, and Search page will likely get to unreasonably large file sizes for efficient loading--which is where a more conventional repository platform that does server-side processing will provide a better experience for your users. There is certainly some optimizations that could be made to the code and data in the template to push the limits, but when you are getting into hundreds of thousands, I doubt the user experience will be good. The build time necessary to generate the collection will also be extremely long for the developer.

I would say CollectionBuilder isn't optimized for extremely large collections, instead it is aimed at smaller more focused collections where presenting the context (as an exhibit or scholarship project) is important. Tens of thousands of items seems more like a database than a collection in this sense.

[hwlong0305]

Thanks so much for this helpful information. We were thinking of building an exhibit that highlights the best images, so perhaps we can use CollectionBuilder for that purpose.