Migrate from LokiJS to LevelDB database

[jakubgs]

Using LokiJS will not scale, and our database has already reached 180 MB:

jakubgs@master-01.do-ams3.ci.release:/docker/ghcmgr/data % ls -lh
total 218M
-rw-r--r-- 1 dockremap dockremap 180M Jan 14 14:17 main.db

And this will only get worse with time. It's probably about time to migrate to SQLite LevelDB.

[jakubgs]

After doing some research I've decided to use LevelDB, as its interface is very simple, as is our usage.

We can use very simple keys like:

const kBuild = ({org, repo, pr, ts, id}) => `${org}!${repo}!${padID(pr)}!${ts}!${padID(id)}`
const kComment = ({org, repo, pr, id}) => `${org}!${repo}!${padID(pr)}!${padID(id)}`

Separated by ! and padded for easy sorting.

[jakubgs]

Wrote a simple migration script with works:

import Loki from 'lokijs'
import { Level } from 'level'

const loadDb = (db) => new Promise(
  (res, rej) => db.loadDatabase({}, err => (err ? rej(err) : res()))
)

const pad = (id, width) => String(id).replace(/[#PR]/g, '').padStart(width, '0')

const kBuild = ({org, repo, pr, ts, id}) => `build!${org}!${repo}!${pad(pr, 6)}!${ts}!${pad(id, 8)}`
const kComment = ({org, repo, pr, id})   => `comment!${org}!${repo}!${pad(pr, 6)}!${pad(id, 12)}`

async function main() {
  const level = new Level('./level.db', { valueEncoding: 'json' })
  const loki = new Loki('ghcmgr.db', {autoload: true, autosave: false})
  await loadDb(loki)

  const builds = loki.getCollection('builds')
  const comments = loki.getCollection('comments')

  for (let build of builds.find()) {
    build.org = 'status-im'
    build.ts = build.meta.created
    delete build['meta']
    delete build['$loki']
    await level.put(kBuild(build), build)
  }

  for (let comment of comments.find()) {
    comment.org = 'status-im'
    comment.id = comment.comment_id
    delete comment['meta']
    delete comment['$loki']
    delete comment['comment_id']
    await level.put(kComment(comment), comment)
  }
}

main().catch((err) => {
  console.error(err)
  process.exit(1)
})

Results in IDs like this

build!status-im!status-go!007283!1768400134086!00000001
build!status-im!status-go!007283!1768400243012!00000001
build!status-im!status-web!000883!1768400605834!00000024

comment!status-im!status-go!007281!003748412405
comment!status-im!status-app!019704!003748438620

And in a DB of size:

[jakubgs@caspair:~/work/github-comment-manager]$ du -hs level.db/
25M	level.db/

[jakubgs]

If I include a simple skip function:

const skipEntry = (obj) => (
  obj.repo == 'status-mobile' ||
  obj.repo == 'status-react' ||
  obj.meta.created < 1609455600000 /* Before 2021 */
)

I can get it down to:

[jakubgs@caspair:~/work/github-comment-manager]$ du -hs level.db/
16M	level.db/