Front matter, YAML, node, markdown, hugo, lotus notes, pdf, sitecore

Minutes with 45,000 pdf attachments have been migrated from Lotus Notes using Hugo

Article 12-10-17 ~5 minutes read · 1023 words

1589 minutes and 45,000 related pdf-attachments are now located as a static site under the main website of Viborg municipality in Denmark, after the municipality has carried out a successful migration of the material, which used to be stored and served by Lotus Notes.

The documents origin from three separate databases, and they take up 32 GB in total. Here the minutes and the many thousand attachments were saved as rich text content in Notes, where the content and links to the pdfs mostly resided in tables. Please refer to Fig. 1.

The old webdesign of the minutes in Viborg Municipality

Fig. 1: One of the minutes from Viborg Municipality as it appeared, when it was served by Lotus Notes/Domino.

The task was to extract all the material in question from Lotus Notes and present it in a cleaner design, while replacing the old Notes-generated tables with more flexible html-code capable of being viewed properly on small screens. In addition, there was a request to expand the design with some «breadcrumbs» at the top, so that users would be able to navigate seamlessly between minutes from different committees and different years.

Ready for launch two weeks later

Beforehand the IT people in Viborg had suspected, that the conversion could take place using the method depicted in this Guide to migration from Notes. At the same time they realised, that JLKM had tested the method in advance and put the results into operation with a website of almost 30,000 documents.

The contact was made, the details were determined and the work could begin. Two weeks later, the converted files were transferred to the defined folder with subfolders on the municipality server, and the result went almost immediately online. The new design is presented in Fig. 2.

The new design of the minutes at Viborg Municipality´s website

Fig. 2: In the new version, navigation has been added at the top, as text elements and icons from the old Notes tables have been repositioned and facelifted.

Metadata was added and old HTML snatch removed

Initially, the PDFs and other data were extracted using Lotus Script and an extraction tool, that was configured to save the documents as html - with the addition of a number of selected categories together with similar metadata at the top.

Then the html documents were converted to markdown format using the js library to-markdown, which features batch conversion on top of Node.js. To-markdown was also able to transfer the programmed metadata from the extract in yaml format to the front matter, which is where markdown stores document info. The js library was also used to provide additional code to make up for some of the page-structure, that had gone with the deleted tables.

The remaining font tags, inline styles and other deprecated html snatch from Notes didn´t survive the final cleanout with regular expressions (regex). Now everything was in place to setup and configure the rebuild using the static page generator Hugo.

Ready for Hugo

Hugo is open source, and the same applies to the far more than 100 themes, which you can choose as the building blocks for a Hugo-website. So the problem is neither price nor to find the right fundament for a Hugo website. The hardest part is actually to choose the theme that best matches your wishes.

On top came out the very minimalistic theme, Goa, which, however, had to go undergo some comprehensive tweaking, until it completely fit in. Among other things, a large number of elements had to be added to the taxonomies, which the category system is named in Hugo. Furthermore the css styling had to be developed almost from scratch in order to pick up some of the abandoned table features in Notes.

Along the way, it turned out that the minutes had added manual links to pdfs in other Notes databases, which we had not anticipated. This was a no-go, as the whole idea is to close Notes down. So the external pdfs were also extracted and transferred to the new repository, as the references to them were converted correspondingly during the exercise of significant amounts of regex gymnastics.

At present the Hugo generated minutes are accessible through an iframe at the Sitecore mainsite of Viborg Municipality, and the combination of the two platforms can be examined in Fig. 3.

Minutes generated by Hugo has come to form a part of Viborg Municipalitys main website

Fig. 3: The converted minutes can be accessed via an iframe on Viborg Municipalitys Sitecore-based website.

Pleased with both results, process and pace

IT Architect Kim Andersen, has been a supervisor of the project, which he has attended on behalf of Viborg Municipality along with web consultant René Petersen. Kim Andersen has turned out to be pleased with the change from Lotus Notes to a static Hugo website:

The migration has obviously led to a general boost of our archived minutes. This applies both to design, where they look considerably more welcoming after they have been pulled out of the tables. Thumbs up also for functionality, as navigation options have increased significantly, he says.

The process is also approved:

I think the migration has gone very smoothly. This, of course, is also due to the fact, that we have migrated to static pages, that are not supposed to be edited subsequently. Accordingly, there has been no need to involve our many editors. This has without doubt made the process considerably leaner. It has turned out, that JLKM was able to commit global design adjustments after imports. This has also proved to be a major advantage.

The migration-project was delivered in a short amount of time, and this is considered an extra bonus by the danish IT Architect.

Even though we offered a wery tight timeframe, JLKM has been able to speed up - and do so quickly. We ended up receiving the requested product both on time and for the agreed price, even though we came up with some suggestions for modifications relatively late in the process. Of course, we can only be extremely satisfied with this. If at some point we need to carry out a similar migration, I would join JLKM again without reservations, says Kim Andersen.

Please don´t hesitate to take a look at the new version of archived minutes at the website of Viborg Municipality (in danish).