Version 11 Content Packaging Changes

With the move to a new underlying database technology in Foundry Virtual Tabletop Version 11, the way that compendium pack data is stored has changed. Previously, all the data for a single pack was stored in a single plain text file with a .db extension, but now each pack exists as its own folder containing several different files, some of which are binary-encoded. This article will provide information on migrating your compendium pack content to the new format, as well as discussing some more advanced tooling like version control.

Glossary of Terms.

NeDB

The old database technology that stored data in .db files.

LevelDB

The new database technology that stores data in folders.

V11 Only

If you only want your content to be available in Foundry Virtual Tabletop Version 11 and newer, this is the simplest migration path. First, you will need to update your module.json or system.json and amend the pack definitions to remove the .db extension from the paths.

{
  "packs": [{
    "name": "monsters",
    "label": "Monsters",
    "system": "some-system",
    "type": "Actor",
    "path": "packs/monsters.db"
  }]
}

{
  "packs": [{
    "name": "monsters",
    "label": "Monsters",
    "system": "some-system",
    "type": "Actor",
    "path": "packs/monsters"
  }]
}

Then, you should launch into a World with your package active, and the core software will take care of migrating your pack to the new format for you. Using the example above, you should see that a new packs/monsters folder has been created in your package directory. Check that everything in the World looks correct and that your package's data is there. If anything went wrong, or you need to repeat the migration, you can leave the World, delete the newly-created folders, and have the core software repeat the migration when you re-launch the World.

Once you are happy that everything is in order, shut down the World so that LevelDB is able to compact its data, then your package can be safely packaged-up and distributed. Be sure to delete the leftover NeDB files so that they do not add needless size to your package. If you are using any sort of version control with your package, you will probably need to make some slight adjustments before you commit the new folders. See the section on version control further down for more information.

V10 Compatibility

The simplest way to maintain backwards-compatibility and allow your content to be used in Version 10 and earlier, and Version 11 of the software, is to continue using the NeDB files. When your package is loaded into a World by a user in Version 11, these pack files will be automatically migrated for them. Whenever your package updates, the package directory is wiped, and your updated packs will be re-migrated. You should take care to not distribute any of the automatically-migrated LevelDB folders with your package in this case.

When opting to continue using NeDB, however, you must consider how modifications to your pack content will be made, since Version 11 will only modify the LevelDB files. The most straight-forward way to do this, is to use a Version 10 instance of the software to make your modifications. This may not always be possible, however, for various reasons. In those cases, you will need to use some external tools to convert your modified LevelDB files back into NeDB files. We provide a command-line application that can do this, see here for details.

Version Control

Line Endings

Particularly when developing on Windows, you may encounter issues related to auto-conversion of line-endings. Some of the LevelDB files are assumed to be text files, and have LF bytes auto-converted to CRLF bytes when checked-out. Not all version control systems attempt to auto-convert line-endings, so this may require no action on your part. If, however, you receive strange errors about not being able to connect to databases, line-endings could be the cause. We provide guidance here on how to resolve this issue if you are using git. If you are using a different version control system, you will need to find out how to disable automatic line-ending conversion in that particular system.

If you are not on Windows, auto-conversion of line-endings is probably not enabled, so you do not need to do this. If you are on Windows, however, you can disable auto-conversion of line-endings by running this command in your project.

git config --local core.autocrlf false

Be aware that this affects all files in your repository, and you may find that you are subsequently accidentally committing CRLF-delimited files to your repository. A safer, more targeted option, is to instruct git specifically that any packs are binary encoded and to not attempt to auto-convert their line-endings. This is a good idea regardless of which operating system you use, as it will mean that anyone else contributing to your project who uses Windows does not run into these issues either. To do this, place a .gitattributes file in the root of your repository, with the following contents:

packs/** binary

This assumes that your compendium packs are stored in the packs/ directory of your repository. If they are stored elsewhere, you should amend the above path. If your packs aren't all stored under the same directory, you can add additional lines, one for each directory. Or if your packs are not stored in a separate directory at all, you can provide the path to the LevelDB folder directly, e.g.:

monsters/** binary
heroes/** binary

Repository Size

Note: The following covers some advanced concepts that are only worth exploring if you use very large packs (several megabytes in size), and modify them very frequently. For the majority of content packs, the following does not apply.

Since the LevelDB files are binary-encoded, git will not attempt to diff them. This means that whenever you commit a change to one of your packs, the entire new state of the pack is stored in your git history, rather than only the diff. Over time this can balloon the size of your repository, particularly if you have very large packs.

One way to address this is by enabling git LFS. This moves large binary files out of your main repository and stores them separately so they can be tracked differently. How to do this will depend on where you host your repositories. GitHub has information on how to configure LFS here, but most git service providers support LFS. If you host your repositories privately, it is still possible to configure LFS on your own server, but it is beyond the scope of this article.

A simpler, more manual solution is to periodically clean your git history of the old copies of these files. Be warned that this is a destructive operation. Once you delete the old copies from your git history, there is no way to retrieve them. You will have only the most recent copy of your pack data available. In most cases, this should not be an issue, but if you find that you often need to roll back your repository and retrieve earlier versions of compendium pack data, then you should not use this method. The BFG repo cleaner is a tool that can be used to purge git history of large binary files. See here for instructions on how you might do that.

The most robust, but more complicated way to address this is to not commit your pack database files to your repository at all. You can store your pack data in some intermediary, human-readable text format in your repository, such as JSON or YAML, and then build a separate archive that contains the LevelDB databases for when your package is installed by Foundry Virtual Tabletop. We provide a command-line application that can unpack LevelDB databases into JSON or YAML, suitable for storing in git repositories, and then re-pack them into a LevelDB database for when your package is ready to be distributed. See here for details on how to use the command-line application.