Blobs and Blobentries
Types
DistributedFactorGraphs.Blobentry
— Typestruct Blobentry
A Blobentry
is a small about of structured data that holds reference information to find an actual blob. Many Blobentry
s can exist on different graph nodes spanning Agents and Factor Graphs which can all reference the same Blob
.
Notes:
blobId
s should be unique within a blobstore and are immutable.
DistributedFactorGraphs.AbstractBlobstore
— TypeAbstractBlobstore{T}
Abstract supertype for all blobstore implementations.
Usage
Subtypes of AbstractBlobstore{T}
must implement the required interface for blob storage and retrieval, such as:
add!(store, blobId, blob)
: Add a new blob to the store.get(store, blobId)
: Retrieve a blob by its ID.list(store)
: List all blob IDs in the store.
The parameter T
represents the type of blobs stored (e.g., Vector{UInt8}
or a custom Blob
type).
See concrete implementations for details.
Design Notes
blobId
is not considered unique across blobstores with different labels only within a single blobstore.- We cannot guarantee that
blobId
is unique across different blobstores with the same label and this is up to the end user. - Within a single blobstore
addBlob!
will fail if there is a UUID collision. - TODO: We should consider using uuid7 for
blobId
s (requires jl v1.12). Blobstrores
are identified by alabel::Symbol
, which allows for multiple blobstores to coexist in the same system.
TODO: If we want to make the blobId
=>Blob pair immutable:
- We can use the tombstone pattern to mark a blob as deleted. See FolderStore in PR#TODO.
Design goal: all Blobstore
s with the same label
can contain the same blobId
=>Blob
pair and the blobs should be identical since they are immutable.
Functions
DistributedFactorGraphs.addBlobentry!
— MethodaddBlobentry!(var, entry)
Add Data Entry to a DFG variable Should be extended if DFG variable is not returned by reference.
Also see: getBlobentry
, addBlob!
, mergeBlobentries!
DistributedFactorGraphs.buildSourceString
— MethodbuildSourceString(dfg, label)
Function to generate source string - agentLabel|graphLabel|varLabel
DistributedFactorGraphs.deleteBlobentry!
— MethoddeleteBlobentry!(var, key)
Delete a blob entry from the factor graph. Note this doesn't remove it from any data stores.
Notes:
- users responsibility to delete data in db before deleting entry
DistributedFactorGraphs.getBlobentries
— MethodgetBlobentries(var)
Get blob entries, Vector{Blobentry}
DistributedFactorGraphs.getBlobentriesVariables
— MethodgetBlobentriesVariables(
dfg,
bLblPattern;
varList,
dropEmpties
)
Get all blob entries matching a Regex pattern over variables
Notes
- Use
dropEmpties=true
to not include empty lists in result. - Use keyword
varList
for which variables to search through.
DistributedFactorGraphs.getBlobentry
— MethodDistributedFactorGraphs.getfirstBlobentry
— MethodgetfirstBlobentry(var, blobId)
Finds and returns the first blob entry that matches the filter.
Also see: getBlobentry
DistributedFactorGraphs.hasBlobentry
— MethodhasBlobentry(var, blobLabel)
Does a blob entry exist with blobLabel
.
DistributedFactorGraphs.incrDataLabelSuffix
— MethodincrDataLabelSuffix(dfg, vla, bllb; datalabel)
If the blob label datalabel
already exists, then this function will return the name datalabel_1
. If the blob label datalabel_1
already exists, then this function will return the name datalabel_2
.
DistributedFactorGraphs.listBlobentries
— MethodlistBlobentries(var)
List the blob entries associated with a particular variable.
DistributedFactorGraphs.listBlobentrySequence
— FunctionlistBlobentrySequence(dfg, lb, pattern)
listBlobentrySequence(dfg, lb, pattern, _sort)
List a collection of blob entries per variable that match a particular pattern::Regex
.
Notes
- Optional sort function argument, default is unsorted.
- Likely use of
sortDFG
for basic Symbol sorting.
- Likely use of
Example
listBlobentrySequence(fg, :x0, r"IMG_CENTER", sortDFG)
15-element Vector{Symbol}:
:IMG_CENTER_21676
:IMG_CENTER_21677
:IMG_CENTER_21678
:IMG_CENTER_21679
...
DistributedFactorGraphs.mergeBlobentries!
— MethodmergeBlobentries!(dst, dlbl, src, slbl, bllb)
Add a blob entry into the destination variable which already exists in a source variable.
See also: addBlobentry!
, getBlobentry
, listBlobentries
, getBlob
DistributedFactorGraphs.mergeBlobentry!
— MethodmergeBlobentry!(var, bde)
Update a Blobentry in the factor graph. If the Blobentry does not exist, it will be added. Notes:
DistributedFactorGraphs.addBlob!
— FunctionAdds a blob to the blob store or dfg with the blobId.
Related addBlobentry!
Implement addBlob!(store::AbstractBlobstore, blobId::UUID, data)
addBlob!(dfg, entry, data)
addBlob!(store, entry, data)
addBlob!(store, data)
addBlob!(store, blobId, data)
addBlob!(store, blobId, data)
addBlob!(store, entry, linkfile)
addBlob!(store, blobId, linkfile)
addBlob!(store, blobId, blob)
addBlob!(store, data, _)
addBlob!(store, blobId, data, _)
defined at deprecated.jl:103
.
DistributedFactorGraphs.deleteBlob!
— FunctionDelete a blob from the blob store or dfg with the given entry.
Related deleteBlobentry!
Implement deleteBlob!(store::AbstractBlobstore, blobId::UUID)
deleteBlob!(dfg, entry)
deleteBlob!(store, entry)
deleteBlob!(store, blobId)
deleteBlob!(store, blobId)
deleteBlob!(store)
deleteBlob!(store, _)
deleteBlob!(store, _)
deleteBlob!(store, blobId)
DistributedFactorGraphs.getBlob
— FunctionGet the data blob for the specified blobstore or dfg.
Related getBlobentry
Implement getBlob(store::AbstractBlobstore, blobId::UUID)
getBlob(dfg, entry)
getBlob(store, entry)
getBlob(store, blobId)
getBlob(store, blobId)
getBlob(store, blobId)
getBlob(store, blobId)
DistributedFactorGraphs.listBlobs
— FunctionList all blobId
s in the blob store. Implement listBlobs(store::AbstractBlobstore)
DistributedFactorGraphs.packBlob
— FunctionpackBlob
Convert a file (JSON, JPG, PNG, BSON, LAS) to Vector{UInt8} for use as a Blob. Returns the blob and MIME type.
DistributedFactorGraphs.unpackBlob
— FunctionunpackBlob
Convert a Blob back to the origanal typ using the MIME type or DataFormat type.
DistributedFactorGraphs.addData!
— FunctionAdd both a Blobentry and Blob to a distributed factor graph or Blobstore. Related addBlobentry!
addData!(dfg, label, entry, blob; hashfunction, checkhash)
addData!(
dfg,
blobstore,
label,
entry,
blob;
hashfunction,
checkhash
)
addData!(
dfg,
blobstorekey,
vLbl,
bLbl,
blob,
timestamp;
kwargs...
)
addData!(dfg, blobstorekey, vLbl, bLbl, blob; ...)
addData!(
dfg,
blobstore,
vLbl,
bLbl,
blob,
timestamp;
description,
metadata,
mimeType,
id,
blobId,
originId,
hashfunction
)
addData!(dfg, blobstore, vLbl, bLbl, blob; ...)
addData!(
dfg,
blobstore,
vLbl,
blobLabel,
blob,
timestamp;
description,
metadata,
mimeType,
origin
)
addData!(dfg, blobstore, vLbl, blobLabel, blob; ...)
DistributedFactorGraphs.deleteData!
— FunctionDelete a blob entry and blob from the blob store or dfg. Related deleteBlobentry!
deleteData!(dfg, vLbl, bLbl)
deleteData!(dfg, blobstore, vLbl, entry)
deleteData!(dfg, blobstore, vLbl, bLbl)
DistributedFactorGraphs.getData
— FunctionGet the blob entry and blob for the specified blobstore or dfg retured as a tuple. Related getBlobentry
getData(dfg, vlabel, key; hashfunction, checkhash, getlast)
getData(
dfg,
blobstore,
label,
key;
hashfunction,
checkhash,
getlast
)
DistributedFactorGraphs.updateData!
— FunctionUpdate a blob entry or blob to the blob store or dfg. Related mergeBlobentry!
updateData!(
dfg,
label,
entry,
blob;
hashfunction,
checkhash
)
updateData!(
dfg,
blobstore,
label,
entry,
blob;
hashfunction
)