From 6e7f0485d8e42278942d64569c19f2b9f504e11a Mon Sep 17 00:00:00 2001 From: Linus Nordberg Date: Tue, 10 Nov 2015 12:44:08 +0100 Subject: Add documentation. --- doc/merge.txt | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 doc/merge.txt diff --git a/doc/merge.txt b/doc/merge.txt new file mode 100644 index 0000000..28757a7 --- /dev/null +++ b/doc/merge.txt @@ -0,0 +1,31 @@ +-*- markdown -*- + +The merge process +================= + +- merge-fetch maintains a single file 'fetched' referring to a given + entry in 'logorder', indicating which entries are fetched and + sequenced so far. + +- merge-backup reads 'fetched' and pushes these entries to secondary + merge nodes, maintaining one file per secondary, + 'backup.', indicating how many entries that have been + copied to and verified at the secondary in question. + +- merge-sth writes a new 'sth' file by reading the + 'backup.' files into a list, picking a new tree size by + sorting the list (in falling order) and indexing it with the + 'backupquorum' config option. If the new tree size is smaller than + what the old 'sth' file says, no new STH is created. + +- merge-dist distributes 'sth' and missing entries to frontend nodes. + +TODO +==== + +- Run the three pieces in parallell. + +- Improve merge-fetch by parallellising it using one process per +storage node writing to a "queue info" file (storage-node, hash) and a +single "queue handling process" reading queue files and writing to the +'fetched' file. -- cgit v1.1