Tracking file changes - provenance

Description

Related to jiras:

(the ePipe depends on the Hops Jira changes)

(this Hopsworks Jira depends on the Hops Jira)

Track file changes in order to generate lineage/provenance information.

Change:

Inode - boolean metaEnabled to a enum MetaStatus(4 values)
MetaStatus:

  • DISABLED

  • META_ENABLED

  • MIN_PROV_ENABLED

  • FULL_PROV_ENABLED

For the first iteration - tracked operations are:
Files:

  • create

  • delete

  • read

  • append

XAttr:

  • add

  • update

  • remove

Using MetaStatus(datasets) to detect to what granularity we track the operations:
MIN_PROV:
file:

  • create

  • delete

xattr:

  • add

  • update

  • remove

FULL_PROV:
all of MIN_PROV
file:

  • read

  • append

Assignee

Alex Ormenisan

Reporter

Alex Ormenisan

Labels

Fix versions

Priority

Medium
Configure