How to Mount S3 Object Storage in a Sandbox

Mount an S3 bucket at a path in a sandbox so that any process reads and writes it like a local directory. A FUSE daemon serves the path, keeps a file cache on the sandbox disk, and uploads changes to the bucket in the background. Each file is one object in the bucket. Paths such as node_modules, .cache, and target stay on the sandbox disk and never reach the bucket.

This guide uses rclone as the FUSE daemon and Latitude.sh Standard object storage in San Jose as the bucket. Freestyle peers directly with Latitude.sh object storage in San Jose: traffic between a sandbox and a San Jose bucket stays on that link instead of crossing the public internet, and a cache-miss round trip is a few milliseconds. Any S3-compatible bucket works with the same commands and a different endpoint.

Create the Bucket and Access Key

In the Latitude.sh dashboard, create a bucket with storage class Standard and region San Jose. The S3 endpoint for that region is s3.us-west-2.storage.sh. Bucket names are global and use lowercase letters, numbers, dots, and hyphens.

Under Access keys, create a key scoped to that bucket with read-write access. The secret is shown once. Keep the key ID and secret in your controller’s secret store; the sandbox receives them in the next step.

Boot a Sandbox and Install the Daemon

Install the SDK in your controller:

pnpm add freestyle@latest

bun add freestyle@latest

npm install freestyle@latest

yarn add freestyle@latest

Boot an Ubuntu sandbox with public egress. The guest kernel has FUSE built in; fuse3 and rclone are not in the base snapshot and are installed here. The rclone install script fetches the current release; Ubuntu’s rclone package also works but is older.

import { Freestyle } from "freestyle";

const freestyle = new Freestyle();

const { vm, vmId } = await freestyle.vms.create({
  snapshotId: "freestyle/ubuntu",
  slug: "s3-workspace",
  firewall: { rules: [{ action: "allow", source: {}, destination: { public: true } }] },
  idleTimeoutSeconds: null,
});

const install = await vm.exec(
  "sudo apt-get update -qq && sudo apt-get install -y -qq fuse3 && " +
    "curl -fsSL https://rclone.org/install.sh | sudo bash && " +
    "echo user_allow_other | sudo tee -a /etc/fuse.conf",
);
if (install.statusCode !== 0) throw new Error(install.stderr);
console.log((await vm.exec("rclone version | head -1")).stdout); // rclone v1.7x.x

user_allow_other lets a mount owned by ubuntu be read by processes running as other users, including root services and Docker containers.

Configure the Remote

Write an rclone config that names the bucket. provider: Other is correct for Latitude.sh. The file is readable only by ubuntu, the user rclone runs as.

const config = `[latitude]
type = s3
provider = Other
access_key_id = ${process.env.LATITUDE_ACCESS_KEY_ID}
secret_access_key = ${process.env.LATITUDE_SECRET_ACCESS_KEY}
endpoint = https://s3.us-west-2.storage.sh
region = us-west-2
acl = private
`;

await vm.exec("mkdir -p /home/ubuntu/.config/rclone");
await vm.fs.writeTextFile("/home/ubuntu/.config/rclone/rclone.conf", config);
await vm.exec("chmod 600 /home/ubuntu/.config/rclone/rclone.conf");

const ls = await vm.exec("rclone lsd latitude:");
if (ls.statusCode !== 0) throw new Error(ls.stderr);
console.log(ls.stdout); // one line per bucket

Replace latitude: with your bucket name in the commands below, in the form latitude:my-bucket.

Sign Requests at the Edge Instead

A TLS rule with an s3 transform signs the sandbox’s storage requests at the edge. The rclone config holds placeholder keys, and the sandbox can reach only one prefix of the bucket. Requests outside the prefix return 403.

Create the rule in your controller:

await freestyle.tls.rules.create({
  action: "allow",
  domain: "s3.us-west-2.storage.sh",
  source: { vmId },
  destination: { public: true },
  transform: [{
    s3: {
      region: "us-west-2",
      credentials: {
        accessKeyId: process.env.LATITUDE_ACCESS_KEY_ID!,
        secretAccessKey: process.env.LATITUDE_SECRET_ACCESS_KEY!,
      },
      scope: {
        bucket: "my-bucket",
        prefix: `workspaces/${vmId}/`,
        operations: [
          "GetObject", "HeadObject", "PutObject", "DeleteObject", "ListObjectsV2",
          "CreateMultipartUpload", "UploadPart", "ListParts",
          "CompleteMultipartUpload", "AbortMultipartUpload", "ListMultipartUploads",
        ],
      },
    },
  }],
});

End the prefix with /: workspaces/vm-1 also matches workspaces/vm-10. rclone uploads files over 200 MiB in parts, which uses the multipart operations.

Write the rclone config with placeholder keys in place of the one above:

const config = `[latitude]
type = s3
provider = Other
access_key_id = placeholder
secret_access_key = placeholder
endpoint = https://s3.us-west-2.storage.sh
region = us-west-2
list_version = 2
no_check_bucket = true
`;

• list_version = 2 lists with ListObjectsV2. Without it, rclone lists with ListObjects v1, which needs ListObjects in the rule.
• no_check_bucket = true stops rclone from checking or creating the bucket before uploads.
• The config has no acl line. The rule rejects ACL headers.

In the mount unit, mount the prefix with a trailing / and add --no-modtime:

ExecStart=/usr/bin/rclone mount latitude:my-bucket/workspaces/<vmId>/ /home/ubuntu/workspace \
  --no-modtime \
  ...

Without the trailing /, rclone sends a HeadObject for workspaces/<vmId>, which is outside the prefix. --no-modtime stops rclone from updating modification times on uploaded files with CopyObject.

The rule rejects CopyObject, which rclone uses for renames. In this mount:

• Renaming a file that has not uploaded yet works.
• Renaming a file that has already uploaded, or renaming any directory, fails with Input/output error.

Use the access key in the rclone config when the workload renames uploaded files or directories.

Mount With a Local Cache

Run the mount as a systemd service so it survives shell exits and starts on boot. --vfs-cache-mode full caches files on the sandbox disk and serves reads and writes from that cache. Without it, random writes and in-place edits fail on S3.

const unit = `[Unit]
Description=S3 workspace mount
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
User=ubuntu
Group=ubuntu
ExecStartPre=/bin/mkdir -p /home/ubuntu/workspace /var/cache/rclone
ExecStart=/usr/bin/rclone mount latitude:my-bucket /home/ubuntu/workspace \\
  --cache-dir /var/cache/rclone \\
  --vfs-cache-mode full \\
  --vfs-cache-max-size 20G \\
  --vfs-cache-max-age 720h \\
  --vfs-write-back 5s \\
  --dir-cache-time 1h \\
  --transfers 16 \\
  --allow-other \\
  --umask 002 \\
  --rc --rc-addr 127.0.0.1:5572 --rc-no-auth \\
  --log-level INFO
ExecStop=/bin/fusermount3 -uz /home/ubuntu/workspace
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target
`;

await vm.fs.writeTextFile("/tmp/s3-workspace.service", unit);
await vm.exec(
  "sudo install -m 644 /tmp/s3-workspace.service /etc/systemd/system/s3-workspace.service && " +
    "sudo mkdir -p /var/cache/rclone && sudo chown ubuntu:ubuntu /var/cache/rclone && " +
    "sudo systemctl daemon-reload && sudo systemctl enable --now s3-workspace",
);

const mounted = await vm.exec("mountpoint /home/ubuntu/workspace");
if (mounted.statusCode !== 0) throw new Error("mount did not come up");

What each group of flags does:

• --cache-dir and --vfs-cache-max-size place the file cache on the sandbox disk and cap it. Size the cap below the free space on the sandbox; the cache evicts least-recently-used files past the cap. --vfs-cache-max-age keeps unused files for 30 days before eviction.
• --vfs-write-back 5s uploads a file five seconds after its last close. A file written and closed repeatedly uploads once per quiet period. A sandbox stopped inside that window keeps the dirty file in /var/cache/rclone, and rclone uploads it the next time the service starts with the same cache directory.
• --dir-cache-time 1h caches directory listings for an hour. S3 has no change notification. Run rclone rc vfs/refresh recursive=true to see changes another writer made to the bucket before the hour is up.
• --rc exposes a local control socket for the checks in the last section.
• Type=notify marks the service started once the mount is up, so units ordered after it see a working mountpoint.

Verify with a round trip through the bucket:

await vm.fs.writeTextFile("/home/ubuntu/workspace/hello.txt", "hello from freestyle\n");
await vm.exec("sleep 8"); // past --vfs-write-back
console.log((await vm.exec("rclone ls latitude:my-bucket")).stdout); // 21 hello.txt

Keep node_modules and Build Output Local

rclone mounts have no per-path sync rule: everything under the mount is bucket-backed. To keep a directory out of the bucket, mount a directory from the sandbox disk on top of it. The mount hides whatever the bucket holds at that path, and every write under it lands on the sandbox disk.

Write a list of local-only paths, relative to the workspace root, and a helper that bind-mounts each one. The helper is idempotent, runs as root, and takes unmount to undo its mounts.

const localOnly = `# One path per line, relative to /home/ubuntu/workspace.
node_modules
app/node_modules
.next
.cache
.turbo
dist
target
.venv
__pycache__
`;
await vm.fs.writeTextFile("/home/ubuntu/.local-only", localOnly);

const helper = `#!/bin/bash
set -euo pipefail
WORKSPACE=/home/ubuntu/workspace
BACKING=/var/lib/local-only
until mountpoint -q "$WORKSPACE"; do sleep 0.2; done
grep -v '^\\s*#' /home/ubuntu/.local-only | grep -v '^\\s*$' | while read -r rel; do
  target="$WORKSPACE/$rel"
  backing="$BACKING/$rel"
  if [ "\${1:-}" = unmount ]; then
    mountpoint -q "$target" && umount -l "$target"
    continue
  fi
  mkdir -p "$target" "$backing"
  chown ubuntu:ubuntu "$backing"
  if ! mountpoint -q "$target"; then
    mount --bind "$backing" "$target"
  fi
done
`;
await vm.fs.writeTextFile("/tmp/local-only-mounts", helper);
await vm.exec(
  "sudo install -m 755 /tmp/local-only-mounts /usr/local/bin/local-only-mounts && " +
    "sudo mkdir -p /var/lib/local-only",
);

Run the helper from a second unit bound to the S3 mount. WantedBy starts it whenever the mount starts, BindsTo stops it when the mount stops, and After orders it behind the mount’s Type=notify readiness.

const localOnlyUnit = `[Unit]
Description=Local-only paths under the S3 workspace
BindsTo=s3-workspace.service
After=s3-workspace.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/bin/local-only-mounts
ExecStop=/usr/local/bin/local-only-mounts unmount

[Install]
WantedBy=s3-workspace.service
`;
await vm.fs.writeTextFile("/tmp/local-only-mounts.service", localOnlyUnit);
await vm.exec(
  "sudo install -m 644 /tmp/local-only-mounts.service /etc/systemd/system/local-only-mounts.service && " +
    "sudo systemctl daemon-reload && sudo systemctl enable local-only-mounts && " +
    "sudo systemctl restart s3-workspace",
);

const bound = await vm.exec("mountpoint /home/ubuntu/workspace/node_modules");
if (bound.statusCode !== 0) throw new Error("local-only mount did not come up");

Confirm a package install stays local:

await vm.exec("cd /home/ubuntu/workspace && npm init -y >/dev/null && npm install --silent left-pad");
await vm.exec("sleep 8");
console.log((await vm.exec("rclone ls latitude:my-bucket")).stdout);
// package.json and package-lock.json are listed; node_modules is not.
console.log((await vm.exec("ls /var/lib/local-only/node_modules")).stdout); // left-pad

Behavior at the boundary:

• rm -rf node_modules removes the contents and fails on the directory itself with Device or resource busy. npm ci, pnpm install, and yarn install clear contents rather than removing the directory and work unchanged. A tool that renames node_modules aside needs the directory removed from .local-only and the bind mount unmounted first.
• S3 has no empty directories, so the mkdir -p on the mount writes nothing to the bucket. The helper recreates each mountpoint on every start.
• Paths that do not exist yet, such as app/node_modules before app is cloned, are created empty by the helper. Rerun sudo local-only-mounts after adding a path to .local-only.
• A path in .local-only that already has contents in the bucket is hidden, not deleted. Unmount it with umount /home/ubuntu/workspace/<path> to see the bucket contents again.

A local-only directory is rebuilt, not restored. On a fresh sandbox against the same bucket, mount, run the helper, then run npm ci from the lockfile in the bucket.

Route Excluded Paths in Your Own Daemon

A daemon you write can apply the rule inside the filesystem instead of by stacking mounts, which survives mv node_modules node_modules.bak and lets a path change from local-only to synced without unmounting. The Rust fuser crate handles the FUSE protocol.

Keep one metadata table for the whole tree, in SQLite on the sandbox disk. Each inode row carries its parent, name, size, mode, mtime, and a local_only flag. Compute the flag at mkdir and create time: a new entry is local-only when its own path matches a rule or its parent is local-only. rename recomputes the flag for the moved subtree and, when a subtree moves from local-only to synced, enqueues its files for upload.

Store each file whole under a cache directory keyed by inode, one object per file. open on a file with no cached copy fetches the object with GetObject and writes it to the cache. read and write go to the cached copy, and write marks the inode dirty. A background uploader drains dirty inodes after a write-back delay and skips any inode whose local_only flag is set; for the rest it uploads the cached file with PutObject. The eviction pass never evicts a dirty file or a local-only file. The cache is the only copy of those.

fn is_local_only(&self, parent: Inode, name: &str) -> bool {
    if self.meta.get(parent).local_only {
        return true;
    }
    let path = self.meta.path_of(parent).join(name);
    self.rules.iter().any(|rule| rule.matches(&path))
}

fn create(&mut self, req: &Request, parent: u64, name: &OsStr, mode: u32, umask: u32, flags: i32, reply: ReplyCreate) {
    let name = name.to_string_lossy();
    let local_only = self.is_local_only(parent, &name);
    let ino = self.meta.insert(parent, &name, mode, local_only);
    reply.created(&TTL, &self.meta.attr(ino), 0, ino, flags as u32);
}

Write the rules file in the same shape as .local-only above: one glob per line, matched against the path relative to the mount root, with ** allowed. Persist the metadata table to the bucket as a single manifest object on a timer and at unmount, and skip local-only rows when writing it, so a fresh sandbox reads the manifest and serves stat and readdir for the whole tree before any file data is fetched.

Flush Before Stopping and Snapshot

A dirty file that has not passed --vfs-write-back exists only in the sandbox cache. Check the upload queue before stopping or snapshotting the sandbox:

for (;;) {
  const stats = await vm.exec("rclone rc vfs/stats");
  const { diskCache } = JSON.parse(stats.stdout);
  if (diskCache.uploadsInProgress + diskCache.uploadsQueued === 0) break;
  await new Promise((resolve) => setTimeout(resolve, 1000));
}

rclone rc vfs/queue lists the queued files with the seconds until each uploads. Stopping the service runs fusermount3 -uz, which detaches the mount and exits rclone. Files still queued stay in /var/cache/rclone and upload when the service next starts with the same cache directory.

A snapshot of the sandbox captures the cache directory, the local-only backing directories, the rclone config, and the running mount. A sandbox created from that snapshot resumes with the mount in place and its node_modules intact. A sandbox created from a plain Ubuntu snapshot against the same bucket repeats the install and mount steps and rebuilds local-only paths.

Two sandboxes writing the same file through the same bucket produce last-upload-wins. Give each sandbox its own prefix, latitude:my-bucket/<vmId>, when sandboxes must not see each other’s writes.