GitHub

Blobs and Blobentries

Types

DistributedFactorGraphs.Blobentry — Type
struct Blobentry

A Blobentry is a small about of structured data that holds reference information to find an actual blob. Many Blobentrys can exist on different graph nodes spanning Agents and Factor Graphs which can all reference the same Blob.

source
DistributedFactorGraphs.verifyBlob — Method
verifyBlob(entry::Blobentry, blob) -> Union{Bool,Nothing}

Checks the integrity of a blob against the hashes stored in the given Blobentry.

  • Verifies the multihash by decoding its algorithm code, recomputing the digest from blob, and comparing it to the stored digest.
  • Additionally verifies the crc32csum checksum.
  • Returns true if all present hashes match.
  • Returns false if any hash does not match.
  • Returns nothing if the multihash algorithm is unregistered.
source
DistributedFactorGraphs.AbstractBlobprovider — Type
AbstractBlobprovider

Abstract supertype for all blob storage implementations in the Virtual Blob System (VBS).

Every concrete subtype represents a physical storage backend and must carry a label::Symbol field (defaulting to :default) that the VBS uses as its routing key. The label also doubles as the provider hint stored in every Blobentry created through this provider.

Required Interface (Layer 1 — Storage Physics)

Subtypes must implement the following Content-Addressable Storage (CAS) primitives, which operate exclusively on raw bytes and Multihashes — no graph logic:

  • putBlob!(provider, m::Multihash, blob::Vector{UInt8}) -> Multihash — Idempotently store blob at key m. (A convenience putBlob!(provider, blob; hash_func) that hashes and delegates is provided.)
  • fetchBlob(provider, multihash) -> Union{Vector{UInt8}, Nothing} — Fetch raw bytes by Multihash, or nothing if absent.
  • hasBlob(provider, multihash) -> Bool — Fast existence / deduplication check.
  • purgeBlob!(provider, multihash) — Physical removal.

Routing & Labelling

Every concrete implementation should accept an optional label keyword in its constructor:

FolderBlobprovider(path::String; label::Symbol = :default)
NavAbilityBlobprovider(;          label::Symbol = :default)
CachedBlobprovider(local, remote; label::Symbol = :default)

Mount providers onto a graph with:

addBlobprovider!(dfg, provider)               # mounts under provider.label
getBlobprovider(dfg, label::Symbol)           # retrieves by label
listBlobproviders(dfg)                        # lists all mounted labels

CAS Design Principles

  • Blobs are immutable and identified by the cryptographic Multihash of their content, not by random UUIDs. This guarantees global deduplication across all providers.
  • putBlob! is idempotent: uploading the same bytes twice is a no-op.
  • Physical blobs are never deleted through the graph metadata API. Only the Blobentry pointer is removed (deleteVariableBlobentry! etc.); garbage collection handles physical cleanup asynchronously.
  • All providers sharing the same content hash hold identical, interchangeable bytes.

Core Implementations

  • FolderBlobprovider — Local disk I/O; acts as a fast ephemeral LRU cache.
  • MemoryBlobprovider — In-memory storage for testing and transient data.
  • CachedBlobprovider — Write-through wrapper composing a local and a remote provider.
source

Functions

DistributedFactorGraphs.getfirstBlobentry — Method
getfirstBlobentry(
    node;
    whereLabel,
    whereMultihash,
    sortby,
    sortlt
)

Finds and returns the first blob entry that matches the filter. The result is sorted by sortby[=getLabel] and sortlt[=natural_lt] before returning the first entry. Also see: getBlobentry

source
DistributedFactorGraphs.incrDataLabelSuffix — Method
incrDataLabelSuffix(dfg, vla, bllb; datalabel)

If the blob label datalabel already exists, then this function will return the name datalabel_1. If the blob label datalabel_1 already exists, then this function will return the name datalabel_2.

source
DistributedFactorGraphs.listBlobentrySequence — Function
listBlobentrySequence(dfg, lb, pattern)
listBlobentrySequence(dfg, lb, pattern, _sort)

List a collection of blob entries per variable that match a particular pattern::Regex.

Notes

  • Optional sort function argument, default is unsorted.
    • Likely use of sortDFG for basic Symbol sorting.

Example

listBlobentrySequence(fg, :x0, r"IMG_CENTER", sortDFG)
15-element Vector{Symbol}:
 :IMG_CENTER_21676
 :IMG_CENTER_21677
 :IMG_CENTER_21678
 :IMG_CENTER_21679
...
source
DistributedFactorGraphs.fetchBlob — Function

Fetch a blob from a Blobprovider. Returns nothing when the multihash is not present (never throws for a missing key).

Related getBlobentry Implement fetchBlob(provider::AbstractBlobprovider, m::Multihash) -> Union{Vector{UInt8}, Nothing}

fetchBlob(provider, m)

defined at /home/runner/work/DistributedFactorGraphs.jl/DistributedFactorGraphs.jl/src/Blobproviders/Blobproviders.jl:77.

fetchBlob(store, m)

defined at /home/runner/work/DistributedFactorGraphs.jl/DistributedFactorGraphs.jl/src/Blobproviders/Blobproviders.jl:116.

fetchBlob(store, m)

defined at /home/runner/work/DistributedFactorGraphs.jl/DistributedFactorGraphs.jl/src/Blobproviders/Blobproviders.jl:166.

fetchBlob(provider, m)

defined at /home/runner/work/DistributedFactorGraphs.jl/DistributedFactorGraphs.jl/src/Blobproviders/Blobproviders.jl:227.

fetchBlob(provider, entry)

defined at /home/runner/work/DistributedFactorGraphs.jl/DistributedFactorGraphs.jl/src/services/blobprovider_ops.jl:154.

source
DistributedFactorGraphs.purgeBlob! — Function
source
DistributedFactorGraphs.putBlob! — Function

Put a blob into the Blobprovider (idempotent CAS write).

The convenience method hashes once and delegates to the core storage method:

putBlob!(provider, blob::Vector{UInt8}; hash_func) -> Multihash   # hashes, delegates
putBlob!(provider, m::Multihash, blob)             -> Multihash   # core — implement this

Related addBlobentry!

putBlob!(provider, m, blob)

defined at /home/runner/work/DistributedFactorGraphs.jl/DistributedFactorGraphs.jl/src/Blobproviders/Blobproviders.jl:81.

putBlob!(store, m, blob)

defined at /home/runner/work/DistributedFactorGraphs.jl/DistributedFactorGraphs.jl/src/Blobproviders/Blobproviders.jl:120.

putBlob!(store, m, blob)

defined at /home/runner/work/DistributedFactorGraphs.jl/DistributedFactorGraphs.jl/src/Blobproviders/Blobproviders.jl:176.

putBlob!(provider, filepath; hash_func)

defined at /home/runner/work/DistributedFactorGraphs.jl/DistributedFactorGraphs.jl/src/Blobproviders/Blobproviders.jl:239.

putBlob!(provider, m, filepath)

defined at /home/runner/work/DistributedFactorGraphs.jl/DistributedFactorGraphs.jl/src/Blobproviders/Blobproviders.jl:250.

putBlob!(provider, blob; hash_func)

defined at /home/runner/work/DistributedFactorGraphs.jl/DistributedFactorGraphs.jl/src/services/blobprovider_ops.jl:159.

source
DistributedFactorGraphs.getDataFormat — Method
getDataFormat(::MIME) -> Union{Type{DataFormat{S}}, Nothing}

Get the FileIO DataFormat for a MIME type. Uses MIMEs.jl and FileIO's extension registry, falls back to _MIMEOverrides.

Returns nothing if no matching format is found.

Examples

getDataFormat(MIME("image/png"))        # format"PNG"
getDataFormat(MIME("application/json")) # format"JSON"
source
DistributedFactorGraphs.getMimetype — Method
getMimetype(::Type{DataFormat{S}}) -> MIME

Get the MIME type for a FileIO DataFormat. Uses FileIO's extension registry and MIMEs.jl for standard types, falls back to _MIMEOverrides for domain-specific formats.

Examples

getMimetype(format"PNG")  # MIME("image/png")
getMimetype(format"JSON") # MIME("application/json")
source
DistributedFactorGraphs.packBlob — Function
packBlob

Convert data to Vector{UInt8} for use as a Blob. Returns (blob, mimetype). The MIME type is automatically determined from the DataFormat.

source
DistributedFactorGraphs.saveAgentBlob! — Function

Save a Blob to a Blobprovider and attach a Blobentry to an agent.

saveAgentBlob!(dfg, agentlabel, blob, entry)

defined at /home/runner/work/DistributedFactorGraphs.jl/DistributedFactorGraphs.jl/src/services/blob_save_load.jl:151.

saveAgentBlob!(dfg, agentlabel, blob, entry_label; ...)
saveAgentBlob!(
    dfg,
    agentlabel,
    blob,
    entry_label,
    provider;
    blobentry_kwargs...
)

defined at /home/runner/work/DistributedFactorGraphs.jl/DistributedFactorGraphs.jl/src/services/blob_save_load.jl:164.

source
DistributedFactorGraphs.saveFactorBlob! — Function

Save a Blob to a Blobprovider and attach a Blobentry to a factor.

saveFactorBlob!(dfg, factor_label, blob, entry)

defined at /home/runner/work/DistributedFactorGraphs.jl/DistributedFactorGraphs.jl/src/services/blob_save_load.jl:191.

saveFactorBlob!(dfg, factor_label, blob, entry_label; ...)
saveFactorBlob!(
    dfg,
    factor_label,
    blob,
    entry_label,
    provider;
    blobentry_kwargs...
)

defined at /home/runner/work/DistributedFactorGraphs.jl/DistributedFactorGraphs.jl/src/services/blob_save_load.jl:204.

source
DistributedFactorGraphs.saveGraphBlob! — Function

Save a Blob to a Blobprovider and attach a Blobentry to a graph.

saveGraphBlob!(dfg, blob, entry)

defined at /home/runner/work/DistributedFactorGraphs.jl/DistributedFactorGraphs.jl/src/services/blob_save_load.jl:117.

saveGraphBlob!(dfg, blob, entry_label; ...)
saveGraphBlob!(
    dfg,
    blob,
    entry_label,
    provider;
    blobentry_kwargs...
)

defined at /home/runner/work/DistributedFactorGraphs.jl/DistributedFactorGraphs.jl/src/services/blob_save_load.jl:125.

source
DistributedFactorGraphs.saveVariableBlob! — Function

Save a Blob to a Blobprovider and attach a Blobentry to a variable.

saveVariableBlob!(dfg, variable_label, blob, entry)

defined at /home/runner/work/DistributedFactorGraphs.jl/DistributedFactorGraphs.jl/src/services/blob_save_load.jl:77.

saveVariableBlob!(
    dfg,
    variable_label,
    blob,
    entry_label;
    ...
)
saveVariableBlob!(
    dfg,
    variable_label,
    blob,
    entry_label,
    provider;
    blobentry_kwargs...
)

defined at /home/runner/work/DistributedFactorGraphs.jl/DistributedFactorGraphs.jl/src/services/blob_save_load.jl:90.

source
DistributedFactorGraphs.CachedBlobprovider — Type
CachedBlobprovider(local, remote; label = :default)

A write-through CAS cache that composes two AbstractBlobproviders.

  • putBlob!: Writes to both remote and local (remote first).
  • fetchBlob: Reads from local first; on miss, fetches from remote and caches locally.
  • purgeBlob!: Purges from both stores.
  • hasBlob: Checks local, falls back to remote.
  • listBlobs: Delegates to remote (the authoritative provider).
source
DistributedFactorGraphs.LinkBlobprovider — Type
LinkBlobprovider(folder; label = :default)

A CAS provider that hardlinks existing files into the standard sharded folder layout (same as FolderBlobprovider). This avoids copying large files while still making them addressable by Multihash.

The specialised putBlob!(provider, filepath::String) streams the file to compute its hash (no full load into RAM), then creates a hardlink at the CAS path. All other Layer 1 verbs (fetchBlob, hasBlob, purgeBlob!, listBlobs) behave identically to FolderBlobprovider.

source
DistributedFactorGraphs.blobfilename — Method
blobfilename(folder, m::Multihash) -> String

Generates the physical file path for a blob using a Git/IPFS style sharding strategy to prevent OS directory-size limits.

source