Persisting Data

If a URCap container application needs to persist data between runs, then one can specify a writable location using the ‘mounts’ field in the manifest.yaml. Given that the application writes data to ‘/foo’, then one should specify the mount as “persistent:/foo”. That is you prepend the desired directory with “persistent:”. For example:

# manifest.yaml
metadata:
  vendorID: universal-robots-test
  urcapID: mount-test
  ...
artifacts:
  containers:
  - id: mounting-test
    image: image.tar
    mounts:
      - mount: "persistent:/foo"
        access: "rw"

Writing files

When writing a file with new content it is recommended to write the data to a temporary file first, and afterward rename the file to the target one. This way the likelihood of ending up with corrupt data in the target file is reduced. The steps are:

create a new temp file
write data to the temp file
rename the temp file with the desired name

Ensuring written data reaches the disk

For performance reasons file writes are not immediately written to physical disk. Instead, they are buffered. This means that sudden power outage might lead to loss of data. If your application requires that a file write reaches disk immediately, then you can reduce the risk of data loss by adding ‘sync’ system calls following your writes. The steps are:

create a new temp file (on the same file system as the target file)
write data to the temp file
fsync() the temp file
rename the temp file with the desired name
fsync() the containing directory

Read more in depth explanation on linux disk synchronization here: https://lwn.net/Articles/457667/

Synchronized writing examples

Here are a few examples of writing data to a file and subsequently asking the OS to persist the data to disk. Error handling has been left out for brevity.

Python

#!/usr/bin/env python3
import os
import uuid
import pathlib

def writeDataToFile(fileName : pathlib.Path, data):
    randomFilePath = fileName.parent / str(uuid.uuid4()) # place tmp file adjacent to target file, so they are on same FS

    with open(randomFilePath, 'w') as f:
        f.write(data)
        os.fsync(f.fileno())

        os.rename(f.name,  fileName)

        dirFd = os.open(fileName.parent, flags=os.O_RDONLY) # get file descriptor of the directory in which 'fileName' resides.
        os.fsync(dirFd)
        os.close(dirFd)

        
path = pathlib.Path("foo/bar")
writeDataToFile(path, "baz")

C++

#include <filesystem>
#include <random>
#include <unistd.h>
#include <fcntl.h>

void writeDataToFile(const std::filesystem::path &outputFile, const void* data, int sz)q
{
    auto randomFilePath = std::filesystem::path{outputFile.parent_path() / std::to_string(rand())};

    int fd = open(randomFilePath.c_str(), O_CREAT | O_WRONLY, 00644);

    write(fd, data, sz);

    fsync(fd);
    close(fd);

    std::filesystem::rename(randomFilePath, outputFile);

    int dirFd = open(outputFile.parent_path() == "" ? "." : outputFile.parent_path().c_str(), O_RDONLY);

    fsync(dirFd);
    close(dirFd);
}

template <typename T>
void writeDataToFile(const std::filesystem::path &outputFile, const std::vector<T> &data){
    writeDataToFile(outputFile, data.data(), data.size() * sizeof(T));
}

int main()
{
    auto data = std::vector<uint16_t>{0xDEAD, 0xBEEF, 0xCAFE, 0xFEED};

    writeDataToFile(std::filesystem::path{"foo/data.txt"}, data);

    return 0;
}

Java

import java.io.File;
import java.io.IOException;
import java.io.FileOutputStream;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.Files;
import java.nio.file.StandardCopyOption;
import java.util.UUID;

class Persistence {
    public static void main(String[] args) {
        writeDataToFile(Paths.get("foo/bar"), "baz");
    }

    private static void writeDataToFile(Path path, String data) {
        Path randomFilePath = Paths.get(path.getParent().toString(), UUID.randomUUID().toString());
        File tmpFile = new File(randomFilePath.toString());

        try (FileOutputStream os = new FileOutputStream(tmpFile)){
            os.write(data.getBytes(), 0, data.length());
            os.getFD().sync();
            Files.move(tmpFile.toPath(), path, StandardCopyOption.REPLACE_EXISTING, StandardCopyOption.ATOMIC_MOVE);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}