Files

How Marfa keeps a file's bytes once, however many items name them, who may read them, and how long they stay.

A file in Marfa comes in two parts. The bytes are a blob, stored once under a hash of their content. The file itself is an item, which names the blob by that hash and carries everything else: a title, a media type, the size of an image.

Keeping them apart stores the same bytes once however many items use them, and keeps an item small enough for every device to hold.

A blob and the item that names it

To add a file, upload its bytes, then create an item that names the hash Marfa answers:

Terminal
marfa blobs upload inventory.txt
stored sha256:7fc0ba0d4e6818ccf1f3eae53ed1ce059625672ff5438d0ab59b12ab6e0305ef (64 bytes, text/plain)

A file item is an item of core.file or a type that inherits from it, such as core.file.image for a photo or core.file.webpage for a saved page. Its blob_ref is the hash, always written sha256: and 64 hex digits:

Request
POST /items
Authorization: Bearer <key>
Content-Type: application/json

{
  "type": "core.file",
  "properties": {
    "title": "Flat inventory",
    "blob_ref": "sha256:7fc0ba0d4e6818ccf1f3eae53ed1ce059625672ff5438d0ab59b12ab6e0305ef",
    "mime_type": "text/plain"
  }
}

Marfa sets the item's size_bytes from the bytes it holds, so no writer has to. The same bytes uploaded again answer the same hash and are stored once, so two items can name one blob. Handle files walks through uploading, downloading and replacing a file, and attaching one to another item.

Find a file by what it says

Marfa reads the text inside files: a PDF's pages, a document's words, a scanned receipt. It writes what it reads onto the file item as extracted_text, where the item holds none, and search matches it. Seconds after the upload above, a search for oak finds the inventory. An image also gains its width and height, and a recording its duration, unless the writer set them. Replacing the bytes clears all four.

The owner can turn text reading off, or OCR on images alone, in Settings.

Who may read a blob

Knowing a hash isn't enough to read the bytes. A credential reads a blob through an item that names it: if you can read a file item, you can read its blob. A hash elsewhere in an item counts too, such as an image embedded in a note's Markdown body.

A reference counts only if whoever wrote it had uploaded the bytes or could already read them, so naming a hash you found somewhere gives you nothing. A file item can't name such a hash at all: the write is refused with 404 blob_not_found. Upload the bytes with the same credential first, as Handle files shows.

A blob you may not read answers 404 blob_not_found, as a missing one does. That holds even for a blob you uploaded, until an item names it.

Download a blob answers the bytes, and takes a Range for part of them. For a page or a player that can't send a credential, marfa blobs url or Get a blob URL answers a signed link that works without one. It lasts an hour by default, and ttl_seconds sets up to seven days.

How long a blob stays

A blob stays as long as something names it, including an item in the bin or an earlier version of an item. Once nothing does, a later housekeeping run purges it, at least blob_cleanup_grace_seconds after it was found unnamed, a day by default, and never while a backup Marfa keeps still names it.

Every instance keeps its blobs on the disk beside the server, and can keep further copies in the backup folder and an object store, as Back up and restore explains.

The rules behind this page are in the blobs and stores chapters of the contract.

On this page