Blobs and Blobentries
Types
DistributedFactorGraphs.Blobentry — Typestruct BlobentryA Blobentry is a small about of structured data that holds reference information to find an actual blob. Many Blobentrys can exist on different graph nodes spanning Agents and Factor Graphs which can all reference the same Blob.
Notes:
blobIds should be unique within a blobstore and are immutable.
DistributedFactorGraphs.AbstractBlobstore — TypeAbstractBlobstore{T}Abstract supertype for all blobstore implementations.
Usage
Subtypes of AbstractBlobstore{T} must implement the required interface for blob storage and retrieval, such as:
add!(store, blobId, blob): Add a new blob to the store.get(store, blobId): Retrieve a blob by its ID.list(store): List all blob IDs in the store.
The parameter T represents the type of blobs stored (e.g., Vector{UInt8} or a custom Blob type).
See concrete implementations for details.
Design Notes
blobIdis not considered unique across blobstores with different labels only within a single blobstore.- We cannot guarantee that
blobIdis unique across different blobstores with the same label and this is up to the end user. - Within a single blobstore
addBlob!will fail if there is a UUID collision. - TODO: We should consider using uuid7 for
blobIds (requires jl v1.12). Blobstroresare identified by alabel::Symbol, which allows for multiple blobstores to coexist in the same system.
TODO: If we want to make the blobId=>Blob pair immutable:
- We can use the tombstone pattern to mark a blob as deleted. See FolderStore in PR#TODO.
Design goal: all Blobstores with the same label can contain the same blobId=>Blob pair and the blobs should be identical since they are immutable.
Functions
DistributedFactorGraphs.addBlobentry! — MethodaddBlobentry!(var, entry)
Add Data Entry to a DFG variable Should be extended if DFG variable is not returned by reference.
Also see: getBlobentry, addBlob!, mergeBlobentries!
DistributedFactorGraphs.buildSourceString — MethodbuildSourceString(dfg, label)
Function to generate source string - agentLabel|graphLabel|varLabel
DistributedFactorGraphs.deleteBlobentry! — MethoddeleteBlobentry!(var, key)
Delete a blob entry from the factor graph. Note this doesn't remove it from any data stores.
Notes:
- users responsibility to delete data in db before deleting entry
DistributedFactorGraphs.getBlobentries — MethodgetBlobentries(var)
Get blob entries, Vector{Blobentry}
DistributedFactorGraphs.getBlobentriesVariables — MethodgetBlobentriesVariables(
dfg,
bLblPattern;
varList,
dropEmpties
)
Get all blob entries matching a Regex pattern over variables
Notes
- Use
dropEmpties=trueto not include empty lists in result. - Use keyword
varListfor which variables to search through.
DistributedFactorGraphs.getBlobentry — MethodDistributedFactorGraphs.getfirstBlobentry — MethodgetfirstBlobentry(var, blobId)
Finds and returns the first blob entry that matches the filter.
Also see: getBlobentry
DistributedFactorGraphs.hasBlobentry — MethodhasBlobentry(var, blobLabel)
Does a blob entry exist with blobLabel.
DistributedFactorGraphs.incrDataLabelSuffix — MethodincrDataLabelSuffix(dfg, vla, bllb; datalabel)
If the blob label datalabel already exists, then this function will return the name datalabel_1. If the blob label datalabel_1 already exists, then this function will return the name datalabel_2.
DistributedFactorGraphs.listBlobentries — MethodlistBlobentries(var)
List the blob entries associated with a particular variable.
DistributedFactorGraphs.listBlobentrySequence — FunctionlistBlobentrySequence(dfg, lb, pattern)
listBlobentrySequence(dfg, lb, pattern, _sort)
List a collection of blob entries per variable that match a particular pattern::Regex.
Notes
- Optional sort function argument, default is unsorted.
- Likely use of
sortDFGfor basic Symbol sorting.
- Likely use of
Example
listBlobentrySequence(fg, :x0, r"IMG_CENTER", sortDFG)
15-element Vector{Symbol}:
:IMG_CENTER_21676
:IMG_CENTER_21677
:IMG_CENTER_21678
:IMG_CENTER_21679
...DistributedFactorGraphs.mergeBlobentries! — MethodmergeBlobentries!(dst, dlbl, src, slbl, bllb)
Add a blob entry into the destination variable which already exists in a source variable.
See also: addBlobentry!, getBlobentry, listBlobentries, getBlob
DistributedFactorGraphs.mergeBlobentry! — MethodmergeBlobentry!(var, bde)
Update a Blobentry in the factor graph. If the Blobentry does not exist, it will be added. Notes:
DistributedFactorGraphs.addBlob! — FunctionAdds a blob to the blob store or dfg with the blobId.
Related addBlobentry! Implement addBlob!(store::AbstractBlobstore, blobId::UUID, data)
addBlob!(dfg, entry, data)addBlob!(store, entry, data)addBlob!(store, data)addBlob!(store, blobId, data)addBlob!(store, blobId, data)addBlob!(store, entry, linkfile)addBlob!(store, blobId, linkfile)addBlob!(store, blobId, blob)addBlob!(store, data, _)
addBlob!(store, blobId, data, _)defined at deprecated.jl:103.
DistributedFactorGraphs.deleteBlob! — FunctionDelete a blob from the blob store or dfg with the given entry.
Related deleteBlobentry! Implement deleteBlob!(store::AbstractBlobstore, blobId::UUID)
deleteBlob!(dfg, entry)deleteBlob!(store, entry)deleteBlob!(store, blobId)deleteBlob!(store, blobId)deleteBlob!(store)deleteBlob!(store, _)deleteBlob!(store, _)deleteBlob!(store, blobId)DistributedFactorGraphs.getBlob — FunctionGet the data blob for the specified blobstore or dfg.
Related getBlobentry Implement getBlob(store::AbstractBlobstore, blobId::UUID)
getBlob(dfg, entry)getBlob(store, entry)getBlob(store, blobId)getBlob(store, blobId)getBlob(store, blobId)getBlob(store, blobId)DistributedFactorGraphs.listBlobs — FunctionList all blobIds in the blob store. Implement listBlobs(store::AbstractBlobstore)
DistributedFactorGraphs.packBlob — FunctionpackBlobConvert a file (JSON, JPG, PNG, BSON, LAS) to Vector{UInt8} for use as a Blob. Returns the blob and MIME type.
DistributedFactorGraphs.unpackBlob — FunctionunpackBlobConvert a Blob back to the origanal typ using the MIME type or DataFormat type.
DistributedFactorGraphs.addData! — FunctionAdd both a Blobentry and Blob to a distributed factor graph or Blobstore. Related addBlobentry!
addData!(dfg, label, entry, blob; hashfunction, checkhash)addData!(
dfg,
blobstore,
label,
entry,
blob;
hashfunction,
checkhash
)addData!(
dfg,
blobstorekey,
vLbl,
bLbl,
blob,
timestamp;
kwargs...
)
addData!(dfg, blobstorekey, vLbl, bLbl, blob; ...)addData!(
dfg,
blobstore,
vLbl,
bLbl,
blob,
timestamp;
description,
metadata,
mimeType,
id,
blobId,
originId,
hashfunction
)
addData!(dfg, blobstore, vLbl, bLbl, blob; ...)addData!(
dfg,
blobstore,
vLbl,
blobLabel,
blob,
timestamp;
description,
metadata,
mimeType,
origin
)
addData!(dfg, blobstore, vLbl, blobLabel, blob; ...)DistributedFactorGraphs.deleteData! — FunctionDelete a blob entry and blob from the blob store or dfg. Related deleteBlobentry!
deleteData!(dfg, vLbl, bLbl)deleteData!(dfg, blobstore, vLbl, entry)deleteData!(dfg, blobstore, vLbl, bLbl)DistributedFactorGraphs.getData — FunctionGet the blob entry and blob for the specified blobstore or dfg retured as a tuple. Related getBlobentry
getData(dfg, vlabel, key; hashfunction, checkhash, getlast)getData(
dfg,
blobstore,
label,
key;
hashfunction,
checkhash,
getlast
)DistributedFactorGraphs.updateData! — FunctionUpdate a blob entry or blob to the blob store or dfg. Related mergeBlobentry!
updateData!(
dfg,
label,
entry,
blob;
hashfunction,
checkhash
)updateData!(
dfg,
blobstore,
label,
entry,
blob;
hashfunction
)