Skip to content

API Reference

Packages:

app.fybrik.io/v1beta1

Resource Types:

Blueprint

↩ Parent

Blueprint is the Schema for the blueprints API

Name Type Description Required
apiVersion string app.fybrik.io/v1beta1 true
kind string Blueprint true
metadata object Refer to the Kubernetes API documentation for the fields of the `metadata` field. true
spec object BlueprintSpec defines the desired state of Blueprint, which defines the components of the workload's data path that run in a particular cluster. In a single cluster environment there is one blueprint per workload (FybrikApplication). In a multi-cluster environment there is one Blueprint per cluster per workload (FybrikApplication).
true
status object BlueprintStatus defines the observed state of Blueprint This includes readiness, error message, and indicators for the Kubernetes resources owned by the Blueprint for cleanup and status monitoring
false

Blueprint.spec

↩ Parent

BlueprintSpec defines the desired state of Blueprint, which defines the components of the workload's data path that run in a particular cluster. In a single cluster environment there is one blueprint per workload (FybrikApplication). In a multi-cluster environment there is one Blueprint per cluster per workload (FybrikApplication).

Name Type Description Required
cluster string Cluster indicates the cluster on which the Blueprint runs
true
modules map[string]object Modules is a map which contains modules that indicate the data path components that run in this cluster The map key is moduleInstanceName which is the unique name for the deployed instance related to this workload
true
modulesNamespace string ModulesNamespace is the namespace where modules should be allocated
true
application object ApplicationContext is a context of the origin FybrikApplication (labels, properties, etc.)
false

Blueprint.spec.modules[key]

↩ Parent

BlueprintModule is a copy of a FybrikModule Custom Resource. It contains the information necessary to instantiate a datapath component, including the parameters relevant for the particular workload.

Name Type Description Required
chart object Chart contains the location of the helm chart with info detailing how to deploy
true
name string Name of the FybrikModule on which this is based
true
arguments object Arguments are the input parameters for a specific instance of a module.
false
assetIds []string assetIDs indicate the assets processed by this module. Included so we can track asset status as well as module status in the future.
false

Blueprint.spec.modules[key].chart

↩ Parent

Chart contains the location of the helm chart with info detailing how to deploy

Name Type Description Required
name string Name of helm chart
true
chartPullSecret string Name of secret containing helm registry credentials
false
values map[string]string Values to pass to helm chart installation
false

Blueprint.spec.modules[key].arguments

↩ Parent

Arguments are the input parameters for a specific instance of a module.

Name Type Description Required
assets []object Assets define asset related arguments, such as data source, transformations, etc.
false

Blueprint.spec.modules[key].arguments.assets[index]

↩ Parent

AssetContext defines the input parameters for modules that access an asset

Name Type Description Required
assetID string AssetID identifies the asset to be used for accessing the data when it is ready It is copied from the FybrikApplication resource
true
capability string Capability of the module
true
args []object List of datastores associated with the asset
false
transformations []object Transformations are different types of processing that may be done to the data as it is copied.
false

Blueprint.spec.modules[key].arguments.assets[index].args[index]

↩ Parent

DataStore contains the details for accessing the data that are sent by catalog connectors Credentials for accessing the data are stored in Vault, in the location represented by Vault property.

Name Type Description Required
connection object Connection has the relevant details for accessing the data (url, table, ssl, etc.)
true
format string Format represents data format (e.g. parquet) as received from catalog connectors
false
vault map[string]object Holds details for retrieving credentials by the modules from Vault store. It is a map so that different credentials can be stored for the different DataFlow operations.
false

Blueprint.spec.modules[key].arguments.assets[index].args[index].connection

↩ Parent

Connection has the relevant details for accessing the data (url, table, ssl, etc.)

Name Type Description Required
name string Name of the connection to the data source
true

Blueprint.spec.modules[key].arguments.assets[index].args[index].vault[key]

↩ Parent

Holds details for retrieving credentials from Vault store.

Name Type Description Required
address string Address is Vault address
true
authPath string AuthPath is the path to auth method i.e. kubernetes
true
role string Role is the Vault role used for retrieving the credentials
true
secretPath string SecretPath is the path of the secret holding the Credentials in Vault
true

Blueprint.spec.modules[key].arguments.assets[index].transformations[index]

↩ Parent

Action to be performed on the data, e.g., masking

Name Type Description Required
name string Action name
true

Blueprint.spec.application

↩ Parent

ApplicationContext is a context of the origin FybrikApplication (labels, properties, etc.)

Name Type Description Required
context object Application context such as intent, role, etc.
false
selector object Application selector is used to identify the user workload. It is obtained from FybrikApplication spec.
false

Blueprint.spec.application.selector

↩ Parent

Application selector is used to identify the user workload. It is obtained from FybrikApplication spec.

Name Type Description Required
matchExpressions []object matchExpressions is a list of label selector requirements. The requirements are ANDed.
false
matchLabels map[string]string matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
false

Blueprint.spec.application.selector.matchExpressions[index]

↩ Parent

A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.

Name Type Description Required
key string key is the label key that the selector applies to.
true
operator string operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
true
values []string values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
false

Blueprint.status

↩ Parent

BlueprintStatus defines the observed state of Blueprint This includes readiness, error message, and indicators for the Kubernetes resources owned by the Blueprint for cleanup and status monitoring

Name Type Description Required
modules map[string]object ModulesState is a map which holds the status of each module its key is the moduleInstanceName which is the unique name for the deployed instance related to this workload
false
observedGeneration integer ObservedGeneration is taken from the Blueprint metadata. This is used to determine during reconcile whether reconcile was called because the desired state changed, or whether status of the allocated resources should be checked.

Format: int64
false
observedState object ObservedState includes information to be reported back to the FybrikApplication resource It includes readiness and error indications, as well as user instructions
false
releases map[string]integer Releases map each release to the observed generation of the blueprint containing this release. At the end of reconcile, each release should be mapped to the latest blueprint version or be uninstalled.
false

Blueprint.status.modules[key]

↩ Parent

ObservedState represents a part of the generated Blueprint/Plotter resource status that allows update of FybrikApplication status

Name Type Description Required
error string Error indicates that there has been an error to orchestrate the modules and provides the error message
false
ready boolean Ready represents that the modules have been orchestrated successfully and the data is ready for usage
false

Blueprint.status.observedState

↩ Parent

ObservedState includes information to be reported back to the FybrikApplication resource It includes readiness and error indications, as well as user instructions

Name Type Description Required
error string Error indicates that there has been an error to orchestrate the modules and provides the error message
false
ready boolean Ready represents that the modules have been orchestrated successfully and the data is ready for usage
false

FybrikApplication

↩ Parent

FybrikApplication provides information about the application whose data is being operated on, the nature of the processing, and the data sets chosen for processing by the application. The FybrikApplication controller obtains instructions regarding any governance related changes that must be performed on the data, identifies the modules capable of performing such changes, and finally generates the Plotter which defines the secure runtime environment and all the components in it. This runtime environment provides the application with access to the data requested in a secure manner and without having to provide any credentials for the data sets. The credentials are obtained automatically by the manager from the credential management system.

Name Type Description Required
apiVersion string app.fybrik.io/v1beta1 true
kind string FybrikApplication true
metadata object Refer to the Kubernetes API documentation for the fields of the `metadata` field. true
spec object FybrikApplicationSpec defines data flows needed by the application, the purpose and other contextual information about the application.
true
status object FybrikApplicationStatus defines the observed state of FybrikApplication.
false

FybrikApplication.spec

↩ Parent

FybrikApplicationSpec defines data flows needed by the application, the purpose and other contextual information about the application.

Name Type Description Required
appInfo object AppInfo contains information describing the reasons for the processing that will be done by the application.
true
data []object Data contains the identifiers of the data to be used by the Data Scientist's application, and the protocol used to access it and the format expected.
true
secretRef string SecretRef points to the secret that holds credentials for each system the user has been authenticated with. The secret is deployed in FybrikApplication namespace.
false
selector object Selector enables to connect the resource to the application Application labels should match the labels in the selector.
false

FybrikApplication.spec.data[index]

↩ Parent

DataContext indicates data set being processed by the workload and includes information about the data format and technologies used to access the data.

Name Type Description Required
dataSetID string DataSetID is a unique identifier of the dataset chosen from the data catalog. For data catalogs that support multiple sub-catalogs, it includes the catalog id and the dataset id. When writing a new dataset it is the name provided by the user or workload generating it.
true
requirements object Requirements from the system
true
flow enum Flows indicates what is being done with the particular dataset - ex: read, write, copy (ingest), delete This is optional for the purpose of backward compatibility. If nothing is provided, read is assumed.

Enum: read, write, delete, copy
false

FybrikApplication.spec.data[index].requirements

↩ Parent

Requirements from the system

Name Type Description Required
flowParams object FlowParams include the requirements for particular data flows
false
interface object Interface indicates the protocol and format expected by the data user
false

FybrikApplication.spec.data[index].requirements.flowParams

↩ Parent

FlowParams include the requirements for particular data flows

Name Type Description Required
catalog string Catalog indicates that the data asset must be cataloged, and in which catalog to register it
false
isNewDataSet boolean IsNewDataSet if true indicates that the DataContext.DataSetID is user provided and not a full catalog / dataset ID. Relevant when writing. A unique ID from the catalog will be provided in the FybrikApplication Status after a new catalog entry is created.
false
metadata object Source asset metadata like asset name, owner, geography, etc Relevant when writing new asset.
false
storageEstimate integer Storage estimate indicates the estimated amount of storage in MB, GB, TB required when writing new data.

Format: int64
false

FybrikApplication.spec.data[index].requirements.flowParams.metadata

↩ Parent

Source asset metadata like asset name, owner, geography, etc Relevant when writing new asset.

Name Type Description Required
columns []object Columns associated with the asset
false
geography string Geography of the resource
false
name string Name of the resource
false
owner string Owner of the resource
false
tags object Tags associated with the asset
false

FybrikApplication.spec.data[index].requirements.flowParams.metadata.columns[index]

↩ Parent

ResourceColumn represents a column in a tabular resource

Name Type Description Required
name string Name of the column
true
tags object Tags associated with the column
false

FybrikApplication.spec.data[index].requirements.interface

↩ Parent

Interface indicates the protocol and format expected by the data user

Name Type Description Required
protocol string Connection type, e.g., S3, Kafka, MySQL
true
dataformat string DataFormat defines the data format type
false

FybrikApplication.spec.selector

↩ Parent

Selector enables to connect the resource to the application Application labels should match the labels in the selector.

Name Type Description Required
workloadSelector object WorkloadSelector enables to connect the resource to the application Application labels should match the labels in the selector.
true
clusterName string Cluster name
false

FybrikApplication.spec.selector.workloadSelector

↩ Parent

WorkloadSelector enables to connect the resource to the application Application labels should match the labels in the selector.

Name Type Description Required
matchExpressions []object matchExpressions is a list of label selector requirements. The requirements are ANDed.
false
matchLabels map[string]string matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
false

FybrikApplication.spec.selector.workloadSelector.matchExpressions[index]

↩ Parent

A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.

Name Type Description Required
key string key is the label key that the selector applies to.
true
operator string operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
true
values []string values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
false

FybrikApplication.status

↩ Parent

FybrikApplicationStatus defines the observed state of FybrikApplication.

Name Type Description Required
assetStates map[string]object AssetStates provides a status per asset
false
errorMessage string ErrorMessage indicates that an error has happened during the reconcile, unrelated to a specific asset
false
generated object Generated resource identifier
false
observedGeneration integer ObservedGeneration is taken from the FybrikApplication metadata. This is used to determine during reconcile whether reconcile was called because the desired state changed, or whether the Blueprint status changed.

Format: int64
false
provisionedStorage map[string]object ProvisionedStorage maps a dataset (identified by AssetID) to the new provisioned bucket. It allows FybrikApplication controller to manage buckets in case the spec has been modified, an error has occurred, or a delete event has been received. ProvisionedStorage has the information required to register the dataset once the owned plotter resource is ready
false
ready boolean Ready is true if all specified assets are either ready to be used or are denied access.
false
validApplication string ValidApplication indicates whether the FybrikApplication is valid given the defined taxonomy
false
validatedGeneration integer ValidatedGeneration is the version of the FyrbikApplication that has been validated with the taxonomy defined.

Format: int64
false

FybrikApplication.status.assetStates[key]

↩ Parent

AssetState defines the observed state of an asset

Name Type Description Required
catalogedAsset string CatalogedAsset provides a new asset identifier after being registered in the enterprise catalog
false
conditions []object Conditions indicate the asset state (Ready, Deny, Error)
false
endpoint object Endpoint provides the endpoint spec from which the asset will be served to the application
false

FybrikApplication.status.assetStates[key].conditions[index]

↩ Parent

Condition describes the state of a FybrikApplication at a certain point.

Name Type Description Required
type string Type of the condition
true
message string Message contains the details of the current condition
false
observedGeneration integer ObservedGeneration is the version of the resource for which the condition has been evaluated

Format: int64
false
status enum Status of the condition, one of (`True`, `False`, `Unknown`).

Enum: True, False, Unknown
Default: Unknown
false

FybrikApplication.status.assetStates[key].endpoint

↩ Parent

Endpoint provides the endpoint spec from which the asset will be served to the application

Name Type Description Required
name string Name of the connection to the data source
true

FybrikApplication.status.generated

↩ Parent

Generated resource identifier

Name Type Description Required
appVersion integer Version of FybrikApplication that has generated this resource

Format: int64
true
kind string Kind of the resource (Blueprint, Plotter)
true
name string Resource name
true
namespace string Resource namespace
true

FybrikApplication.status.provisionedStorage[key]

↩ Parent

DatasetDetails holds details of the provisioned storage

Name Type Description Required
datasetRef string deprecated
false
details object Dataset information
false
persistent boolean Persistent storage (not to be removed after FybrikApplication is deleted)
false
resourceMetadata object Resource Metadata
false
secretRef object Reference to a secret where the credentials are stored
false

FybrikApplication.status.provisionedStorage[key].details

↩ Parent

Dataset information

Name Type Description Required
connection object Connection has the relevant details for accessing the data (url, table, ssl, etc.)
true
format string Format represents data format (e.g. parquet) as received from catalog connectors
false
vault map[string]object Holds details for retrieving credentials by the modules from Vault store. It is a map so that different credentials can be stored for the different DataFlow operations.
false

FybrikApplication.status.provisionedStorage[key].details.connection

↩ Parent

Connection has the relevant details for accessing the data (url, table, ssl, etc.)

Name Type Description Required
name string Name of the connection to the data source
true

FybrikApplication.status.provisionedStorage[key].details.vault[key]

↩ Parent

Holds details for retrieving credentials from Vault store.

Name Type Description Required
address string Address is Vault address
true
authPath string AuthPath is the path to auth method i.e. kubernetes
true
role string Role is the Vault role used for retrieving the credentials
true
secretPath string SecretPath is the path of the secret holding the Credentials in Vault
true

FybrikApplication.status.provisionedStorage[key].resourceMetadata

↩ Parent

Resource Metadata

Name Type Description Required
columns []object Columns associated with the asset
false
geography string Geography of the resource
false
name string Name of the resource
false
owner string Owner of the resource
false
tags object Tags associated with the asset
false

FybrikApplication.status.provisionedStorage[key].resourceMetadata.columns[index]

↩ Parent

ResourceColumn represents a column in a tabular resource

Name Type Description Required
name string Name of the column
true
tags object Tags associated with the column
false

FybrikApplication.status.provisionedStorage[key].secretRef

↩ Parent

Reference to a secret where the credentials are stored

Name Type Description Required
name string Name
true
namespace string Namespace
true

FybrikModule

↩ Parent

FybrikModule is a description of an injectable component. the parameters it requires, as well as the specification of how to instantiate such a component. It is used as metadata only. There is no status nor reconciliation.

Name Type Description Required
apiVersion string app.fybrik.io/v1beta1 true
kind string FybrikModule true
metadata object Refer to the Kubernetes API documentation for the fields of the `metadata` field. true
spec object FybrikModuleSpec contains the info common to all modules, which are one of the components that process, load, write, audit, monitor the data used by the data scientist's application.
true
status object FybrikModuleStatus defines the observed state of FybrikModule.
false

FybrikModule.spec

↩ Parent

FybrikModuleSpec contains the info common to all modules, which are one of the components that process, load, write, audit, monitor the data used by the data scientist's application.

Name Type Description Required
capabilities []object Capabilities declares what this module knows how to do and the types of data it knows how to handle The key to the map is a CapabilityType string
true
chart object Reference to a Helm chart that allows deployment of the resources required for this module
true
type string May be one of service, config or plugin Service: Means that the control plane deploys the component that performs the capability Config: Another pre-installed service performs the capability and the module deployed configures it for the particular workload or dataset Plugin: Indicates that this module performs a capability as part of another service or module rather than as a stand-alone module
true
dependencies []object Other components that must be installed in order for this module to work
false
description string An explanation of what this module does
false
pluginType string Plugin type indicates the plugin technology used to invoke the capabilities Ex: vault, fybrik-wasm... Should be provided if type is plugin
false
statusIndicators []object StatusIndicators allow to check status of a non-standard resource that can not be computed by helm/kstatus
false

FybrikModule.spec.capabilities[index]

↩ Parent

Capability declares what this module knows how to do and the types of data it knows how to handle

Name Type Description Required
capability string Capability declares what this module knows how to do - ex: read, write, transform...
true
actions []object Actions are the data transformations that the module supports
false
api object API indicates to the application how to access the capabilities provided by the module
false
plugins []object Plugins enable the module to add libraries to perform actions rather than implementing them by itself
false
scope enum Scope indicates at what level the capability is used: workload, asset, cluster If not indicated it is assumed to be asset

Enum: asset, workload, cluster
false
supportedInterfaces []object Copy should have one or more instances in the list, and its content should have source and sink Read should have one or more instances in the list, each with source populated Write should have one or more instances in the list, each with sink populated This field may not be required if not handling data
false

FybrikModule.spec.capabilities[index].actions[index]

↩ Parent

Name Type Description Required
name string Unique name of an action supported by the module
true

FybrikModule.spec.capabilities[index].api

↩ Parent

API indicates to the application how to access the capabilities provided by the module

Name Type Description Required
connection object Connection information
true
dataFormat string Data format
false

FybrikModule.spec.capabilities[index].api.connection

↩ Parent

Connection information

Name Type Description Required
name string Name of the connection to the data source
true

FybrikModule.spec.capabilities[index].plugins[index]

↩ Parent

Name Type Description Required
dataFormat string DataFormat indicates the format of data the plugin knows how to process
true
pluginType string PluginType indicates the technology used for the module and the plugin to interact The values supported should come from the module taxonomy Examples of such mechanisms are vault plugins, wasm, etc
true

FybrikModule.spec.capabilities[index].supportedInterfaces[index]

↩ Parent

ModuleInOut specifies the protocol and format of the data input and output by the module - if any

Name Type Description Required
sink object Sink specifies the output data protocol and format
false
source object Source specifies the input data protocol and format
false

FybrikModule.spec.capabilities[index].supportedInterfaces[index].sink

↩ Parent

Sink specifies the output data protocol and format

Name Type Description Required
protocol string Connection type, e.g., S3, Kafka, MySQL
true
dataformat string DataFormat defines the data format type
false

FybrikModule.spec.capabilities[index].supportedInterfaces[index].source

↩ Parent

Source specifies the input data protocol and format

Name Type Description Required
protocol string Connection type, e.g., S3, Kafka, MySQL
true
dataformat string DataFormat defines the data format type
false

FybrikModule.spec.chart

↩ Parent

Reference to a Helm chart that allows deployment of the resources required for this module

Name Type Description Required
name string Name of helm chart
true
chartPullSecret string Name of secret containing helm registry credentials
false
values map[string]string Values to pass to helm chart installation
false

FybrikModule.spec.dependencies[index]

↩ Parent

Dependency details another component on which this module relies - i.e. a pre-requisit

Name Type Description Required
name string Name is the name of the dependent component
true
type enum Type provides information used in determining how to instantiate the component

Enum: module, connector, feature
true

FybrikModule.spec.statusIndicators[index]

↩ Parent

ResourceStatusIndicator is used to determine the status of an orchestrated resource

Name Type Description Required
kind string Kind provides information about the resource kind
true
successCondition string SuccessCondition specifies a condition that indicates that the resource is ready It uses kubernetes label selection syntax (https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)
true
errorMessage string ErrorMessage specifies the resource field to check for an error, e.g. status.errorMsg
false
failureCondition string FailureCondition specifies a condition that indicates the resource failure It uses kubernetes label selection syntax (https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)
false

FybrikModule.status

↩ Parent

FybrikModuleStatus defines the observed state of FybrikModule.

Name Type Description Required
conditions []object Conditions indicate the module states with respect to validation
false

FybrikModule.status.conditions[index]

↩ Parent

Condition describes the state of a FybrikApplication at a certain point.

Name Type Description Required
type string Type of the condition
true
message string Message contains the details of the current condition
false
observedGeneration integer ObservedGeneration is the version of the resource for which the condition has been evaluated

Format: int64
false
status enum Status of the condition, one of (`True`, `False`, `Unknown`).

Enum: True, False, Unknown
Default: Unknown
false

FybrikStorageAccount

↩ Parent

FybrikStorageAccount defines a storage account used for copying data. Only S3 based storage is supported. It contains endpoint, region and a reference to the credentials a Owner of the asset is responsible to store the credentials

Name Type Description Required
apiVersion string app.fybrik.io/v1beta1 true
kind string FybrikStorageAccount true
metadata object Refer to the Kubernetes API documentation for the fields of the `metadata` field. true
spec object FybrikStorageAccountSpec defines the desired state of FybrikStorageAccount
true
status object FybrikStorageAccountStatus defines the observed state of FybrikStorageAccount
false

FybrikStorageAccount.spec

↩ Parent

FybrikStorageAccountSpec defines the desired state of FybrikStorageAccount

Name Type Description Required
endpoint string Endpoint for accessing the data
true
id string Identification of a storage account
true
region string Storage region
true
secretRef string A name of k8s secret deployed in the control plane. This secret includes secretKey and accessKey credentials for S3 bucket
true

Plotter

↩ Parent

Plotter is the Schema for the plotters API

Name Type Description Required
apiVersion string app.fybrik.io/v1beta1 true
kind string Plotter true
metadata object Refer to the Kubernetes API documentation for the fields of the `metadata` field. true
spec object PlotterSpec defines the desired state of Plotter, which is applied in a multi-clustered environment. Plotter declares what needs to be installed and where (as blueprints running on remote clusters) which provides the Data Scientist's application with secure and governed access to the data requested in the FybrikApplication.
true
status object PlotterStatus defines the observed state of Plotter This includes readiness, error message, and indicators received from blueprint resources owned by the Plotter for cleanup and status monitoring
false

Plotter.spec

↩ Parent

PlotterSpec defines the desired state of Plotter, which is applied in a multi-clustered environment. Plotter declares what needs to be installed and where (as blueprints running on remote clusters) which provides the Data Scientist's application with secure and governed access to the data requested in the FybrikApplication.

Name Type Description Required
assets map[string]object Assets is a map holding information about the assets The key is the assetID
true
flows []object
true
modulesNamespace string ModulesNamespace is the namespace where modules should be allocated
true
templates map[string]object Templates is a map holding the templates used in this plotter steps The key is the template name
true
appInfo object Application context to be transferred to the modules
false
appSelector object Selector enables to connect the resource to the application Application labels should match the labels in the selector. For some flows the selector may not be used.
false

Plotter.spec.assets[key]

↩ Parent

AssetDetails is a list of assets used in the fybrikapplication. In addition to assets declared in fybrikapplication, AssetDetails list also contains assets that are allocated by the control-plane in order to serve fybrikapplication

Name Type Description Required
assetDetails object DataStore contains the details for accessing the data that are sent by catalog connectors Credentials for accessing the data are stored in Vault, in the location represented by Vault property.
true
advertisedAssetId string AdvertisedAssetID links this asset to asset from fybrikapplication and is used by user facing services
false

Plotter.spec.assets[key].assetDetails

↩ Parent

DataStore contains the details for accessing the data that are sent by catalog connectors Credentials for accessing the data are stored in Vault, in the location represented by Vault property.

Name Type Description Required
connection object Connection has the relevant details for accessing the data (url, table, ssl, etc.)
true
format string Format represents data format (e.g. parquet) as received from catalog connectors
false
vault map[string]object Holds details for retrieving credentials by the modules from Vault store. It is a map so that different credentials can be stored for the different DataFlow operations.
false

Plotter.spec.assets[key].assetDetails.connection

↩ Parent

Connection has the relevant details for accessing the data (url, table, ssl, etc.)

Name Type Description Required
name string Name of the connection to the data source
true

Plotter.spec.assets[key].assetDetails.vault[key]

↩ Parent

Holds details for retrieving credentials from Vault store.

Name Type Description Required
address string Address is Vault address
true
authPath string AuthPath is the path to auth method i.e. kubernetes
true
role string Role is the Vault role used for retrieving the credentials
true
secretPath string SecretPath is the path of the secret holding the Credentials in Vault
true

Plotter.spec.flows[index]

↩ Parent

Flows is the list of data flows driven from fybrikapplication: Each element in the list holds the flow of the data requested in fybrikapplication.

Name Type Description Required
assetId string AssetID indicates the data set being used in this data flow
true
flowType enum Type of the flow (e.g. read)

Enum: read, write, delete, copy
true
name string Name of the flow
true
subFlows []object
true

Plotter.spec.flows[index].subFlows[index]

↩ Parent

Subflows is a list of data flows which are originated from the same data asset but are triggered differently (e.g., one upon init trigger and one upon workload trigger)

Name Type Description Required
flowType enum Type of the flow (e.g. read)

Enum: read, write, delete, copy
true
name string Name of the SubFlow
true
steps [][]object Steps defines a series of sequential/parallel data flow steps The first dimension represents parallel data flows. The second sequential components within the same parallel data flow.
true
triggers []enum Triggers
true

Plotter.spec.flows[index].subFlows[index].steps[index][index]

↩ Parent

DataFlowStep contains details on a single data flow step

Name Type Description Required
cluster string Name of the cluster this step is executed on
true
name string Name of the step
true
template string Template is the name of the template to execute the step The full details of the template can be extracted from Plotter.spec.templates list field.
true
parameters object Step parameters TODO why not flatten the parameters into this data flow step
false

Plotter.spec.flows[index].subFlows[index].steps[index][index].parameters

↩ Parent

Step parameters TODO why not flatten the parameters into this data flow step

Name Type Description Required
action []object Actions are the data transformations that the module supports
false
api object ResourceDetails includes asset connection details
false
args []object
false

Plotter.spec.flows[index].subFlows[index].steps[index][index].parameters.action[index]

↩ Parent

Action to be performed on the data, e.g., masking

Name Type Description Required
name string Action name
true

Plotter.spec.flows[index].subFlows[index].steps[index][index].parameters.api

↩ Parent

ResourceDetails includes asset connection details

Name Type Description Required
connection object Connection information
true
dataFormat string Data format
false

Plotter.spec.flows[index].subFlows[index].steps[index][index].parameters.api.connection

↩ Parent

Connection information

Name Type Description Required
name string Name of the connection to the data source
true

Plotter.spec.flows[index].subFlows[index].steps[index][index].parameters.args[index]

↩ Parent

StepArgument describes a step: it could be assetID or an endpoint of another step

Name Type Description Required
api object API holds information for accessing a module instance
false
assetId string AssetID identifies the source asset of this step
false

Plotter.spec.flows[index].subFlows[index].steps[index][index].parameters.args[index].api

↩ Parent

API holds information for accessing a module instance

Name Type Description Required
connection object Connection information
true
dataFormat string Data format
false

Plotter.spec.flows[index].subFlows[index].steps[index][index].parameters.args[index].api.connection

↩ Parent

Connection information

Name Type Description Required
name string Name of the connection to the data source
true

Plotter.spec.templates[key]

↩ Parent

Template contains basic information about the required modules to serve the fybrikapplication e.g., the module helm chart name.

Name Type Description Required
modules []object Modules is a list of dependent modules. e.g., if a plugin module is used then the service module is used in should appear first in the modules list of the same template. If the modules list contains more than one module, the first module in the list is referred to as the "primary module" of which all the parameters to this template are sent to.
true
name string Name of the template
false

Plotter.spec.templates[key].modules[index]

↩ Parent

ModuleInfo is a copy of FybrikModule Custom Resource. It contains information to instantiate resource of type FybrikModule.

Name Type Description Required
capability string Module capability
true
chart object Chart contains the information needed to use helm to install the capability
true
name string Name of the module
true
type string May be one of service, config or plugin Service: Means that the control plane deploys the component that performs the capability Config: Another pre-installed service performs the capability and the module deployed configures it for the particular workload or dataset Plugin: Indicates that this module performs a capability as part of another service or module rather than as a stand-alone module
true
scope enum Scope indicates at what level the capability is used: workload, asset, cluster If not indicated it is assumed to be asset

Enum: asset, workload, cluster
false

Plotter.spec.templates[key].modules[index].chart

↩ Parent

Chart contains the information needed to use helm to install the capability

Name Type Description Required
name string Name of helm chart
true
chartPullSecret string Name of secret containing helm registry credentials
false
values map[string]string Values to pass to helm chart installation
false

Plotter.spec.appSelector

↩ Parent

Selector enables to connect the resource to the application Application labels should match the labels in the selector. For some flows the selector may not be used.

Name Type Description Required
workloadSelector object WorkloadSelector enables to connect the resource to the application Application labels should match the labels in the selector.
true
clusterName string Cluster name
false

Plotter.spec.appSelector.workloadSelector

↩ Parent

WorkloadSelector enables to connect the resource to the application Application labels should match the labels in the selector.

Name Type Description Required
matchExpressions []object matchExpressions is a list of label selector requirements. The requirements are ANDed.
false
matchLabels map[string]string matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
false

Plotter.spec.appSelector.workloadSelector.matchExpressions[index]

↩ Parent

A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.

Name Type Description Required
key string key is the label key that the selector applies to.
true
operator string operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
true
values []string values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
false

Plotter.status

↩ Parent

PlotterStatus defines the observed state of Plotter This includes readiness, error message, and indicators received from blueprint resources owned by the Plotter for cleanup and status monitoring

Name Type Description Required
assets map[string]object Assets is a map containing the status per asset. The key of this map is assetId
false
blueprints map[string]object
false
conditions []object Conditions represent the possible error and failure conditions
false
flows map[string]object Flows is a map containing the status for each flow the key is the flow name
false
observedGeneration integer ObservedGeneration is taken from the Plotter metadata. This is used to determine during reconcile whether reconcile was called because the desired state changed, or whether status of the allocated blueprints should be checked.

Format: int64
false
observedState object ObservedState includes information to be reported back to the FybrikApplication resource It includes readiness and error indications, as well as user instructions
false
readyTimestamp string

Format: date-time
false

Plotter.status.assets[key]

↩ Parent

ObservedState represents a part of the generated Blueprint/Plotter resource status that allows update of FybrikApplication status

Name Type Description Required
error string Error indicates that there has been an error to orchestrate the modules and provides the error message
false
ready boolean Ready represents that the modules have been orchestrated successfully and the data is ready for usage
false

Plotter.status.blueprints[key]

↩ Parent

MetaBlueprint defines blueprint metadata (name, namespace) and status

Name Type Description Required
name string
true
namespace string
true
status object BlueprintStatus defines the observed state of Blueprint This includes readiness, error message, and indicators for the Kubernetes resources owned by the Blueprint for cleanup and status monitoring
true

Plotter.status.blueprints[key].status

↩ Parent

BlueprintStatus defines the observed state of Blueprint This includes readiness, error message, and indicators for the Kubernetes resources owned by the Blueprint for cleanup and status monitoring

Name Type Description Required
modules map[string]object ModulesState is a map which holds the status of each module its key is the moduleInstanceName which is the unique name for the deployed instance related to this workload
false
observedGeneration integer ObservedGeneration is taken from the Blueprint metadata. This is used to determine during reconcile whether reconcile was called because the desired state changed, or whether status of the allocated resources should be checked.

Format: int64
false
observedState object ObservedState includes information to be reported back to the FybrikApplication resource It includes readiness and error indications, as well as user instructions
false
releases map[string]integer Releases map each release to the observed generation of the blueprint containing this release. At the end of reconcile, each release should be mapped to the latest blueprint version or be uninstalled.
false

Plotter.status.blueprints[key].status.modules[key]

↩ Parent

ObservedState represents a part of the generated Blueprint/Plotter resource status that allows update of FybrikApplication status

Name Type Description Required
error string Error indicates that there has been an error to orchestrate the modules and provides the error message
false
ready boolean Ready represents that the modules have been orchestrated successfully and the data is ready for usage
false

Plotter.status.blueprints[key].status.observedState

↩ Parent

ObservedState includes information to be reported back to the FybrikApplication resource It includes readiness and error indications, as well as user instructions

Name Type Description Required
error string Error indicates that there has been an error to orchestrate the modules and provides the error message
false
ready boolean Ready represents that the modules have been orchestrated successfully and the data is ready for usage
false

Plotter.status.conditions[index]

↩ Parent

Condition describes the state of a FybrikApplication at a certain point.

Name Type Description Required
type string Type of the condition
true
message string Message contains the details of the current condition
false
observedGeneration integer ObservedGeneration is the version of the resource for which the condition has been evaluated

Format: int64
false
status enum Status of the condition, one of (`True`, `False`, `Unknown`).

Enum: True, False, Unknown
Default: Unknown
false

Plotter.status.flows[key]

↩ Parent

FlowStatus includes information to be reported back to the FybrikApplication resource It holds the status per data flow

Name Type Description Required
subFlows map[string]object
true
status object ObservedState includes information about the current flow It includes readiness and error indications, as well as user instructions
false

Plotter.status.flows[key].subFlows[key]

↩ Parent

ObservedState represents a part of the generated Blueprint/Plotter resource status that allows update of FybrikApplication status

Name Type Description Required
error string Error indicates that there has been an error to orchestrate the modules and provides the error message
false
ready boolean Ready represents that the modules have been orchestrated successfully and the data is ready for usage
false

Plotter.status.flows[key].status

↩ Parent

ObservedState includes information about the current flow It includes readiness and error indications, as well as user instructions

Name Type Description Required
error string Error indicates that there has been an error to orchestrate the modules and provides the error message
false
ready boolean Ready represents that the modules have been orchestrated successfully and the data is ready for usage
false

Plotter.status.observedState

↩ Parent

ObservedState includes information to be reported back to the FybrikApplication resource It includes readiness and error indications, as well as user instructions

Name Type Description Required
error string Error indicates that there has been an error to orchestrate the modules and provides the error message
false
ready boolean Ready represents that the modules have been orchestrated successfully and the data is ready for usage
false

app.fybrik.io/v1beta2

Resource Types:

FybrikStorageAccount

↩ Parent

FybrikStorageAccount is a storage account Fybrik uses to dynamically allocate space for datasets whose creation or copy it orchestrates.

Name Type Description Required
apiVersion string app.fybrik.io/v1beta2 true
kind string FybrikStorageAccount true
metadata object Refer to the Kubernetes API documentation for the fields of the `metadata` field. true
spec object FybrikStorageAccountSpec defines the desired state of FybrikStorageAccount
true
status object FybrikStorageAccountStatus defines the observed state of FybrikStorageAccount
false

FybrikStorageAccount.spec

↩ Parent

FybrikStorageAccountSpec defines the desired state of FybrikStorageAccount

Name Type Description Required
geography string Storage geography
true
id string Identification of a storage account
true
secretRef string A name of k8s secret deployed in the control plane.
true
type string Type of the storage, e.g., s3
true

katalog.fybrik.io/v1alpha1

Resource Types:

Asset

↩ Parent

Asset defines an asset in the catalog

Name Type Description Required
apiVersion string katalog.fybrik.io/v1alpha1 true
kind string Asset true
metadata object Refer to the Kubernetes API documentation for the fields of the `metadata` field. true
spec object
true

Asset.spec

↩ Parent

Name Type Description Required
details object Asset details
true
metadata object Asset metadata
true
secretRef object Reference to a Secret resource holding credentials for this asset
true

Asset.spec.details

↩ Parent

Asset details

Name Type Description Required
connection object Connection information
true
dataFormat string Data format
false

Asset.spec.details.connection

↩ Parent

Connection information

Name Type Description Required
name string Name of the connection to the data source
true

Asset.spec.metadata

↩ Parent

Asset metadata

Name Type Description Required
columns []object Columns associated with the asset
false
geography string Geography of the resource
false
name string Name of the resource
false
owner string Owner of the resource
false
tags object Tags associated with the asset
false

Asset.spec.metadata.columns[index]

↩ Parent

ResourceColumn represents a column in a tabular resource

Name Type Description Required
name string Name of the column
true
tags object Tags associated with the column
false

Asset.spec.secretRef

↩ Parent

Reference to a Secret resource holding credentials for this asset

Name Type Description Required
name string Name of the Secret resource
true
namespace string Namespace of the Secret resource. If it is empty then the asset namespace is used.
false