API Reference
Packages:
app.fybrik.io/v1beta1
Resource Types:
Blueprint
Blueprint is the Schema for the blueprints API
Name | Type | Description | Required |
---|---|---|---|
apiVersion | string | app.fybrik.io/v1beta1 | true |
kind | string | Blueprint | true |
metadata | object | Refer to the Kubernetes API documentation for the fields of the `metadata` field. | true |
spec | object |
BlueprintSpec defines the desired state of Blueprint, which defines the components of the workload's data path that run in a particular cluster. In a single cluster environment there is one blueprint per workload (FybrikApplication). In a multi-cluster environment there is one Blueprint per cluster per workload (FybrikApplication). |
true |
status | object |
BlueprintStatus defines the observed state of Blueprint This includes readiness, error message, and indicators for the Kubernetes resources owned by the Blueprint for cleanup and status monitoring |
false |
Blueprint.spec
BlueprintSpec defines the desired state of Blueprint, which defines the components of the workload's data path that run in a particular cluster. In a single cluster environment there is one blueprint per workload (FybrikApplication). In a multi-cluster environment there is one Blueprint per cluster per workload (FybrikApplication).
Name | Type | Description | Required |
---|---|---|---|
cluster | string |
Cluster indicates the cluster on which the Blueprint runs |
true |
modules | map[string]object |
Modules is a map which contains modules that indicate the data path components that run in this cluster The map key is moduleInstanceName which is the unique name for the deployed instance related to this workload |
true |
modulesNamespace | string |
ModulesNamespace is the namespace where modules should be allocated |
true |
application | object |
ApplicationContext is a context of the origin FybrikApplication (labels, properties, etc.) |
false |
Blueprint.spec.modules[key]
BlueprintModule is a copy of a FybrikModule Custom Resource. It contains the information necessary to instantiate a datapath component, including the parameters relevant for the particular workload.
Name | Type | Description | Required |
---|---|---|---|
chart | object |
Chart contains the location of the helm chart with info detailing how to deploy |
true |
name | string |
Name of the FybrikModule on which this is based |
true |
arguments | object |
Arguments are the input parameters for a specific instance of a module. |
false |
assetIds | []string |
assetIDs indicate the assets processed by this module. Included so we can track asset status as well as module status in the future. |
false |
Blueprint.spec.modules[key].chart
Chart contains the location of the helm chart with info detailing how to deploy
Name | Type | Description | Required |
---|---|---|---|
name | string |
Name of helm chart |
true |
chartPullSecret | string |
Name of secret containing helm registry credentials |
false |
values | map[string]string |
Values to pass to helm chart installation |
false |
Blueprint.spec.modules[key].arguments
Arguments are the input parameters for a specific instance of a module.
Name | Type | Description | Required |
---|---|---|---|
assets | []object |
Assets define asset related arguments, such as data source, transformations, etc. |
false |
Blueprint.spec.modules[key].arguments.assets[index]
AssetContext defines the input parameters for modules that access an asset
Name | Type | Description | Required |
---|---|---|---|
assetID | string |
AssetID identifies the asset to be used for accessing the data when it is ready It is copied from the FybrikApplication resource |
true |
capability | string |
Capability of the module |
true |
args | []object |
List of datastores associated with the asset |
false |
transformations | []object |
Transformations are different types of processing that may be done to the data as it is copied. |
false |
Blueprint.spec.modules[key].arguments.assets[index].args[index]
DataStore contains the details for accesing the data that are sent by catalog connectors Credentials for accesing the data are stored in Vault, in the location represented by Vault property.
Name | Type | Description | Required |
---|---|---|---|
connection | object |
Connection has the relevant details for accesing the data (url, table, ssl, etc.) |
true |
format | string |
Format represents data format (e.g. parquet) as received from catalog connectors |
false |
vault | map[string]object |
Holds details for retrieving credentials by the modules from Vault store. It is a map so that different credentials can be stored for the different DataFlow operations. |
false |
Blueprint.spec.modules[key].arguments.assets[index].args[index].connection
Connection has the relevant details for accesing the data (url, table, ssl, etc.)
Name | Type | Description | Required |
---|---|---|---|
name | string |
Name of the connection to the data source |
true |
Blueprint.spec.modules[key].arguments.assets[index].args[index].vault[key]
Holds details for retrieving credentials from Vault store.
Name | Type | Description | Required |
---|---|---|---|
address | string |
Address is Vault address |
true |
authPath | string |
AuthPath is the path to auth method i.e. kubernetes |
true |
role | string |
Role is the Vault role used for retrieving the credentials |
true |
secretPath | string |
SecretPath is the path of the secret holding the Credentials in Vault |
true |
Blueprint.spec.modules[key].arguments.assets[index].transformations[index]
Name | Type | Description | Required |
---|---|---|---|
name | string |
|
true |
Blueprint.spec.application
ApplicationContext is a context of the origin FybrikApplication (labels, properties, etc.)
Name | Type | Description | Required |
---|---|---|---|
context | object |
Application context such as intent, role, etc. |
false |
selector | object |
Application selector is used to identify the user workload. It is obtained from FybrikApplication spec. |
false |
Blueprint.spec.application.selector
Application selector is used to identify the user workload. It is obtained from FybrikApplication spec.
Name | Type | Description | Required |
---|---|---|---|
matchExpressions | []object |
matchExpressions is a list of label selector requirements. The requirements are ANDed. |
false |
matchLabels | map[string]string |
matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. |
false |
Blueprint.spec.application.selector.matchExpressions[index]
A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
Name | Type | Description | Required |
---|---|---|---|
key | string |
key is the label key that the selector applies to. |
true |
operator | string |
operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. |
true |
values | []string |
values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. |
false |
Blueprint.status
BlueprintStatus defines the observed state of Blueprint This includes readiness, error message, and indicators for the Kubernetes resources owned by the Blueprint for cleanup and status monitoring
Name | Type | Description | Required |
---|---|---|---|
modules | map[string]object |
ModulesState is a map which holds the status of each module its key is the moduleInstanceName which is the unique name for the deployed instance related to this workload |
false |
observedGeneration | integer |
ObservedGeneration is taken from the Blueprint metadata. This is used to determine during reconcile whether reconcile was called because the desired state changed, or whether status of the allocated resources should be checked. Format: int64 |
false |
observedState | object |
ObservedState includes information to be reported back to the FybrikApplication resource It includes readiness and error indications, as well as user instructions |
false |
releases | map[string]integer |
Releases map each release to the observed generation of the blueprint containing this release. At the end of reconcile, each release should be mapped to the latest blueprint version or be uninstalled. |
false |
Blueprint.status.modules[key]
ObservedState represents a part of the generated Blueprint/Plotter resource status that allows update of FybrikApplication status
Name | Type | Description | Required |
---|---|---|---|
error | string |
Error indicates that there has been an error to orchestrate the modules and provides the error message |
false |
ready | boolean |
Ready represents that the modules have been orchestrated successfully and the data is ready for usage |
false |
Blueprint.status.observedState
ObservedState includes information to be reported back to the FybrikApplication resource It includes readiness and error indications, as well as user instructions
Name | Type | Description | Required |
---|---|---|---|
error | string |
Error indicates that there has been an error to orchestrate the modules and provides the error message |
false |
ready | boolean |
Ready represents that the modules have been orchestrated successfully and the data is ready for usage |
false |
FybrikApplication
FybrikApplication provides information about the application whose data is being operated on, the nature of the processing, and the data sets chosen for processing by the application. The FybrikApplication controller obtains instructions regarding any governance related changes that must be performed on the data, identifies the modules capable of performing such changes, and finally generates the Plotter which defines the secure runtime environment and all the components in it. This runtime environment provides the application with access to the data requested in a secure manner and without having to provide any credentials for the data sets. The credentials are obtained automatically by the manager from the credential management system.
Name | Type | Description | Required |
---|---|---|---|
apiVersion | string | app.fybrik.io/v1beta1 | true |
kind | string | FybrikApplication | true |
metadata | object | Refer to the Kubernetes API documentation for the fields of the `metadata` field. | true |
spec | object |
FybrikApplicationSpec defines data flows needed by the application, the purpose and other contextual information about the application. |
true |
status | object |
FybrikApplicationStatus defines the observed state of FybrikApplication. |
false |
FybrikApplication.spec
FybrikApplicationSpec defines data flows needed by the application, the purpose and other contextual information about the application.
Name | Type | Description | Required |
---|---|---|---|
appInfo | object |
AppInfo contains information describing the reasons for the processing that will be done by the application. |
true |
data | []object |
Data contains the identifiers of the data to be used by the Data Scientist's application, and the protocol used to access it and the format expected. |
true |
secretRef | string |
SecretRef points to the secret that holds credentials for each system the user has been authenticated with. The secret is deployed in FybrikApplication namespace. |
false |
selector | object |
Selector enables to connect the resource to the application Application labels should match the labels in the selector. |
false |
FybrikApplication.spec.data[index]
DataContext indicates data set being processed by the workload and includes information about the data format and technologies used to access the data.
Name | Type | Description | Required |
---|---|---|---|
dataSetID | string |
DataSetID is a unique identifier of the dataset chosen from the data catalog. For data catalogs that support multiple sub-catalogs, it includes the catalog id and the dataset id. When writing a new dataset it is the name provided by the user or workload generating it. |
true |
requirements | object |
Requirements from the system |
true |
flow | enum |
Flows indicates what is being done with the particular dataset - ex: read, write, copy (ingest), delete This is optional for the purpose of backward compatibility. If nothing is provided, read is assumed. Enum: read, write, delete, copy |
false |
FybrikApplication.spec.data[index].requirements
Requirements from the system
Name | Type | Description | Required |
---|---|---|---|
flowParams | object |
FlowParams include the requirements for particular data flows |
false |
interface | object |
Interface indicates the protocol and format expected by the data user |
false |
FybrikApplication.spec.data[index].requirements.flowParams
FlowParams include the requirements for particular data flows
Name | Type | Description | Required |
---|---|---|---|
catalog | string |
Catalog indicates that the data asset must be cataloged, and in which catalog to register it |
false |
isNewDataSet | boolean |
IsNewDataSet if true indicates that the DataContext.DataSetID is user provided and not a full catalog / dataset ID. Relevant when writing. A unique ID from the catalog will be provided in the FybrikApplication Status after a new catalog entry is created. |
false |
metadata | object |
Source asset metadata like asset name, owner, geography, etc Relevant when writing new asset. |
false |
storageEstimate | integer |
Storage estimate indicates the estimated amount of storage in MB, GB, TB required when writing new data. Format: int64 |
false |
FybrikApplication.spec.data[index].requirements.flowParams.metadata
Source asset metadata like asset name, owner, geography, etc Relevant when writing new asset.
Name | Type | Description | Required |
---|---|---|---|
columns | []object |
Columns associated with the asset |
false |
geography | string |
Geography of the resource |
false |
name | string |
Name of the resource |
false |
owner | string |
Owner of the resource |
false |
tags | object |
Tags associated with the asset |
false |
FybrikApplication.spec.data[index].requirements.flowParams.metadata.columns[index]
ResourceColumn represents a column in a tabular resource
Name | Type | Description | Required |
---|---|---|---|
name | string |
Name of the column |
true |
tags | object |
Tags associated with the column |
false |
FybrikApplication.spec.data[index].requirements.interface
Interface indicates the protocol and format expected by the data user
Name | Type | Description | Required |
---|---|---|---|
protocol | string |
Protocol defines the interface protocol used for data transactions |
true |
dataformat | string |
DataFormat defines the data format type |
false |
FybrikApplication.spec.selector
Selector enables to connect the resource to the application Application labels should match the labels in the selector.
Name | Type | Description | Required |
---|---|---|---|
workloadSelector | object |
WorkloadSelector enables to connect the resource to the application Application labels should match the labels in the selector. |
true |
clusterName | string |
Cluster name |
false |
FybrikApplication.spec.selector.workloadSelector
WorkloadSelector enables to connect the resource to the application Application labels should match the labels in the selector.
Name | Type | Description | Required |
---|---|---|---|
matchExpressions | []object |
matchExpressions is a list of label selector requirements. The requirements are ANDed. |
false |
matchLabels | map[string]string |
matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. |
false |
FybrikApplication.spec.selector.workloadSelector.matchExpressions[index]
A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
Name | Type | Description | Required |
---|---|---|---|
key | string |
key is the label key that the selector applies to. |
true |
operator | string |
operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. |
true |
values | []string |
values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. |
false |
FybrikApplication.status
FybrikApplicationStatus defines the observed state of FybrikApplication.
Name | Type | Description | Required |
---|---|---|---|
assetStates | map[string]object |
AssetStates provides a status per asset |
false |
errorMessage | string |
ErrorMessage indicates that an error has happened during the reconcile, unrelated to a specific asset |
false |
generated | object |
Generated resource identifier |
false |
observedGeneration | integer |
ObservedGeneration is taken from the FybrikApplication metadata. This is used to determine during reconcile whether reconcile was called because the desired state changed, or whether the Blueprint status changed. Format: int64 |
false |
provisionedStorage | map[string]object |
ProvisionedStorage maps a dataset (identified by AssetID) to the new provisioned bucket. It allows FybrikApplication controller to manage buckets in case the spec has been modified, an error has occurred, or a delete event has been received. ProvisionedStorage has the information required to register the dataset once the owned plotter resource is ready |
false |
ready | boolean |
Ready is true if all specified assets are either ready to be used or are denied access. |
false |
validApplication | string |
ValidApplication indicates whether the FybrikApplication is valid given the defined taxonomy |
false |
validatedGeneration | integer |
ValidatedGeneration is the version of the FyrbikApplication that has been validated with the taxonomy defined. Format: int64 |
false |
FybrikApplication.status.assetStates[key]
AssetState defines the observed state of an asset
Name | Type | Description | Required |
---|---|---|---|
catalogedAsset | string |
CatalogedAsset provides a new asset identifier after being registered in the enterprise catalog |
false |
conditions | []object |
Conditions indicate the asset state (Ready, Deny, Error) |
false |
endpoint | object |
Endpoint provides the endpoint spec from which the asset will be served to the application |
false |
FybrikApplication.status.assetStates[key].conditions[index]
Condition describes the state of a FybrikApplication at a certain point.
Name | Type | Description | Required |
---|---|---|---|
type | string |
Type of the condition |
true |
message | string |
Message contains the details of the current condition |
false |
observedGeneration | integer |
ObservedGeneration is the version of the resource for which the condition has been evaluated Format: int64 |
false |
status | enum |
Status of the condition, one of (`True`, `False`, `Unknown`). Enum: True, False, Unknown Default: Unknown |
false |
FybrikApplication.status.assetStates[key].endpoint
Endpoint provides the endpoint spec from which the asset will be served to the application
Name | Type | Description | Required |
---|---|---|---|
name | string |
Name of the connection to the data source |
true |
FybrikApplication.status.generated
Generated resource identifier
Name | Type | Description | Required |
---|---|---|---|
appVersion | integer |
Version of FybrikApplication that has generated this resource Format: int64 |
true |
kind | string |
Kind of the resource (Blueprint, Plotter) |
true |
name | string |
Name of the resource |
true |
namespace | string |
Namespace of the resource |
true |
FybrikApplication.status.provisionedStorage[key]
DatasetDetails holds details of the provisioned storage
Name | Type | Description | Required |
---|---|---|---|
datasetRef | string |
Reference to a Dataset resource containing the request to provision storage |
false |
details | object |
Dataset information |
false |
resourceMetadata | object |
Resource Metadata |
false |
secretRef | object |
Reference to a secret where the credentials are stored |
false |
FybrikApplication.status.provisionedStorage[key].details
Dataset information
Name | Type | Description | Required |
---|---|---|---|
connection | object |
Connection has the relevant details for accesing the data (url, table, ssl, etc.) |
true |
format | string |
Format represents data format (e.g. parquet) as received from catalog connectors |
false |
vault | map[string]object |
Holds details for retrieving credentials by the modules from Vault store. It is a map so that different credentials can be stored for the different DataFlow operations. |
false |
FybrikApplication.status.provisionedStorage[key].details.connection
Connection has the relevant details for accesing the data (url, table, ssl, etc.)
Name | Type | Description | Required |
---|---|---|---|
name | string |
Name of the connection to the data source |
true |
FybrikApplication.status.provisionedStorage[key].details.vault[key]
Holds details for retrieving credentials from Vault store.
Name | Type | Description | Required |
---|---|---|---|
address | string |
Address is Vault address |
true |
authPath | string |
AuthPath is the path to auth method i.e. kubernetes |
true |
role | string |
Role is the Vault role used for retrieving the credentials |
true |
secretPath | string |
SecretPath is the path of the secret holding the Credentials in Vault |
true |
FybrikApplication.status.provisionedStorage[key].resourceMetadata
Resource Metadata
Name | Type | Description | Required |
---|---|---|---|
columns | []object |
Columns associated with the asset |
false |
geography | string |
Geography of the resource |
false |
name | string |
Name of the resource |
false |
owner | string |
Owner of the resource |
false |
tags | object |
Tags associated with the asset |
false |
FybrikApplication.status.provisionedStorage[key].resourceMetadata.columns[index]
ResourceColumn represents a column in a tabular resource
Name | Type | Description | Required |
---|---|---|---|
name | string |
Name of the column |
true |
tags | object |
Tags associated with the column |
false |
FybrikApplication.status.provisionedStorage[key].secretRef
Reference to a secret where the credentials are stored
Name | Type | Description | Required |
---|---|---|---|
name | string |
Secret name |
true |
namespace | string |
Secret Namespace |
true |
FybrikModule
FybrikModule is a description of an injectable component. the parameters it requires, as well as the specification of how to instantiate such a component. It is used as metadata only. There is no status nor reconciliation.
Name | Type | Description | Required |
---|---|---|---|
apiVersion | string | app.fybrik.io/v1beta1 | true |
kind | string | FybrikModule | true |
metadata | object | Refer to the Kubernetes API documentation for the fields of the `metadata` field. | true |
spec | object |
FybrikModuleSpec contains the info common to all modules, which are one of the components that process, load, write, audit, monitor the data used by the data scientist's application. |
true |
status | object |
FybrikModuleStatus defines the observed state of FybrikModule. |
false |
FybrikModule.spec
FybrikModuleSpec contains the info common to all modules, which are one of the components that process, load, write, audit, monitor the data used by the data scientist's application.
Name | Type | Description | Required |
---|---|---|---|
capabilities | []object |
Capabilities declares what this module knows how to do and the types of data it knows how to handle The key to the map is a CapabilityType string |
true |
chart | object |
Reference to a Helm chart that allows deployment of the resources required for this module |
true |
type | string |
May be one of service, config or plugin Service: Means that the control plane deploys the component that performs the capability Config: Another pre-installed service performs the capability and the module deployed configures it for the particular workload or dataset Plugin: Indicates that this module performs a capability as part of another service or module rather than as a stand-alone module |
true |
dependencies | []object |
Other components that must be installed in order for this module to work |
false |
description | string |
An explanation of what this module does |
false |
pluginType | string |
Plugin type indicates the plugin technology used to invoke the capabilities Ex: vault, fybrik-wasm... Should be provided if type is plugin |
false |
statusIndicators | []object |
StatusIndicators allow to check status of a non-standard resource that can not be computed by helm/kstatus |
false |
FybrikModule.spec.capabilities[index]
Capability declares what this module knows how to do and the types of data it knows how to handle
Name | Type | Description | Required |
---|---|---|---|
capability | string |
Capability declares what this module knows how to do - ex: read, write, transform... |
true |
actions | []object |
Actions are the data transformations that the module supports |
false |
api | object |
API indicates to the application how to access the capabilities provided by the module |
false |
plugins | []object |
Plugins enable the module to add libraries to perform actions rather than implementing them by itself |
false |
scope | enum |
Scope indicates at what level the capability is used: workload, asset, cluster If not indicated it is assumed to be asset Enum: asset, workload, cluster |
false |
supportedInterfaces | []object |
Copy should have one or more instances in the list, and its content should have source and sink Read should have one or more instances in the list, each with source populated Write should have one or more instances in the list, each with sink populated This field may not be required if not handling data |
false |
FybrikModule.spec.capabilities[index].actions[index]
Name | Type | Description | Required |
---|---|---|---|
name | string |
Unique name of an action supported by the module |
true |
FybrikModule.spec.capabilities[index].api
API indicates to the application how to access the capabilities provided by the module
Name | Type | Description | Required |
---|---|---|---|
connection | object |
Connection information |
true |
dataFormat | string |
Data format |
false |
FybrikModule.spec.capabilities[index].api.connection
Connection information
Name | Type | Description | Required |
---|---|---|---|
name | string |
Name of the connection to the data source |
true |
FybrikModule.spec.capabilities[index].plugins[index]
Name | Type | Description | Required |
---|---|---|---|
dataFormat | string |
DataFormat indicates the format of data the plugin knows how to process |
true |
pluginType | string |
PluginType indicates the technology used for the module and the plugin to interact The values supported should come from the module taxonomy Examples of such mechanisms are vault plugins, wasm, etc |
true |
FybrikModule.spec.capabilities[index].supportedInterfaces[index]
ModuleInOut specifies the protocol and format of the data input and output by the module - if any
FybrikModule.spec.capabilities[index].supportedInterfaces[index].sink
Sink specifies the output data protocol and format
Name | Type | Description | Required |
---|---|---|---|
protocol | string |
Protocol defines the interface protocol used for data transactions |
true |
dataformat | string |
DataFormat defines the data format type |
false |
FybrikModule.spec.capabilities[index].supportedInterfaces[index].source
Source specifies the input data protocol and format
Name | Type | Description | Required |
---|---|---|---|
protocol | string |
Protocol defines the interface protocol used for data transactions |
true |
dataformat | string |
DataFormat defines the data format type |
false |
FybrikModule.spec.chart
Reference to a Helm chart that allows deployment of the resources required for this module
Name | Type | Description | Required |
---|---|---|---|
name | string |
Name of helm chart |
true |
chartPullSecret | string |
Name of secret containing helm registry credentials |
false |
values | map[string]string |
Values to pass to helm chart installation |
false |
FybrikModule.spec.dependencies[index]
Dependency details another component on which this module relies - i.e. a pre-requisit
Name | Type | Description | Required |
---|---|---|---|
name | string |
Name is the name of the dependent component |
true |
type | enum |
Type provides information used in determining how to instantiate the component Enum: module, connector, feature |
true |
FybrikModule.spec.statusIndicators[index]
ResourceStatusIndicator is used to determine the status of an orchestrated resource
Name | Type | Description | Required |
---|---|---|---|
kind | string |
Kind provides information about the resource kind |
true |
successCondition | string |
SuccessCondition specifies a condition that indicates that the resource is ready It uses kubernetes label selection syntax (https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/) |
true |
errorMessage | string |
ErrorMessage specifies the resource field to check for an error, e.g. status.errorMsg |
false |
failureCondition | string |
FailureCondition specifies a condition that indicates the resource failure It uses kubernetes label selection syntax (https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/) |
false |
FybrikModule.status
FybrikModuleStatus defines the observed state of FybrikModule.
Name | Type | Description | Required |
---|---|---|---|
conditions | []object |
Conditions indicate the module states with respect to validation |
false |
FybrikModule.status.conditions[index]
Condition describes the state of a FybrikApplication at a certain point.
Name | Type | Description | Required |
---|---|---|---|
type | string |
Type of the condition |
true |
message | string |
Message contains the details of the current condition |
false |
observedGeneration | integer |
ObservedGeneration is the version of the resource for which the condition has been evaluated Format: int64 |
false |
status | enum |
Status of the condition, one of (`True`, `False`, `Unknown`). Enum: True, False, Unknown Default: Unknown |
false |
FybrikStorageAccount
FybrikStorageAccount defines a storage account used for copying data. Only S3 based storage is supported. It contains endpoint, region and a reference to the credentials a Owner of the asset is responsible to store the credentials
Name | Type | Description | Required |
---|---|---|---|
apiVersion | string | app.fybrik.io/v1beta1 | true |
kind | string | FybrikStorageAccount | true |
metadata | object | Refer to the Kubernetes API documentation for the fields of the `metadata` field. | true |
spec | object |
FybrikStorageAccountSpec defines the desired state of FybrikStorageAccount |
true |
status | object |
FybrikStorageAccountStatus defines the observed state of FybrikStorageAccount |
false |
FybrikStorageAccount.spec
FybrikStorageAccountSpec defines the desired state of FybrikStorageAccount
Name | Type | Description | Required |
---|---|---|---|
endpoint | string |
Endpoint for accessing the data |
true |
id | string |
Identification of a storage account |
true |
region | string |
Storage region |
true |
secretRef | string |
A name of k8s secret deployed in the control plane. This secret includes secretKey and accessKey credentials for S3 bucket |
true |
Plotter
Plotter is the Schema for the plotters API
Name | Type | Description | Required |
---|---|---|---|
apiVersion | string | app.fybrik.io/v1beta1 | true |
kind | string | Plotter | true |
metadata | object | Refer to the Kubernetes API documentation for the fields of the `metadata` field. | true |
spec | object |
PlotterSpec defines the desired state of Plotter, which is applied in a multi-clustered environment. Plotter declares what needs to be installed and where (as blueprints running on remote clusters) which provides the Data Scientist's application with secure and governed access to the data requested in the FybrikApplication. |
true |
status | object |
PlotterStatus defines the observed state of Plotter This includes readiness, error message, and indicators received from blueprint resources owned by the Plotter for cleanup and status monitoring |
false |
Plotter.spec
PlotterSpec defines the desired state of Plotter, which is applied in a multi-clustered environment. Plotter declares what needs to be installed and where (as blueprints running on remote clusters) which provides the Data Scientist's application with secure and governed access to the data requested in the FybrikApplication.
Name | Type | Description | Required |
---|---|---|---|
assets | map[string]object |
Assets is a map holding information about the assets The key is the assetID |
true |
flows | []object |
|
true |
modulesNamespace | string |
ModulesNamespace is the namespace where modules should be allocated |
true |
templates | map[string]object |
Templates is a map holding the templates used in this plotter steps The key is the template name |
true |
appInfo | object |
Application context to be transferred to the modules |
false |
appSelector | object |
Selector enables to connect the resource to the application Application labels should match the labels in the selector. For some flows the selector may not be used. |
false |
Plotter.spec.assets[key]
AssetDetails is a list of assets used in the fybrikapplication. In addition to assets declared in fybrikapplication, AssetDetails list also contains assets that are allocated by the control-plane in order to serve fybrikapplication
Name | Type | Description | Required |
---|---|---|---|
assetDetails | object |
DataStore contains the details for accesing the data that are sent by catalog connectors Credentials for accesing the data are stored in Vault, in the location represented by Vault property. |
true |
advertisedAssetId | string |
AdvertisedAssetID links this asset to asset from fybrikapplication and is used by user facing services |
false |
Plotter.spec.assets[key].assetDetails
DataStore contains the details for accesing the data that are sent by catalog connectors Credentials for accesing the data are stored in Vault, in the location represented by Vault property.
Name | Type | Description | Required |
---|---|---|---|
connection | object |
Connection has the relevant details for accesing the data (url, table, ssl, etc.) |
true |
format | string |
Format represents data format (e.g. parquet) as received from catalog connectors |
false |
vault | map[string]object |
Holds details for retrieving credentials by the modules from Vault store. It is a map so that different credentials can be stored for the different DataFlow operations. |
false |
Plotter.spec.assets[key].assetDetails.connection
Connection has the relevant details for accesing the data (url, table, ssl, etc.)
Name | Type | Description | Required |
---|---|---|---|
name | string |
Name of the connection to the data source |
true |
Plotter.spec.assets[key].assetDetails.vault[key]
Holds details for retrieving credentials from Vault store.
Name | Type | Description | Required |
---|---|---|---|
address | string |
Address is Vault address |
true |
authPath | string |
AuthPath is the path to auth method i.e. kubernetes |
true |
role | string |
Role is the Vault role used for retrieving the credentials |
true |
secretPath | string |
SecretPath is the path of the secret holding the Credentials in Vault |
true |
Plotter.spec.flows[index]
Flows is the list of data flows driven from fybrikapplication: Each element in the list holds the flow of the data requested in fybrikapplication.
Name | Type | Description | Required |
---|---|---|---|
assetId | string |
AssetID indicates the data set being used in this data flow |
true |
flowType | enum |
Type of the flow (e.g. read) Enum: read, write, delete, copy |
true |
name | string |
Name of the flow |
true |
subFlows | []object |
|
true |
Plotter.spec.flows[index].subFlows[index]
Subflows is a list of data flows which are originated from the same data asset but are triggered differently (e.g., one upon init trigger and one upon workload trigger)
Name | Type | Description | Required |
---|---|---|---|
flowType | enum |
Type of the flow (e.g. read) Enum: read, write, delete, copy |
true |
name | string |
Name of the SubFlow |
true |
steps | [][]object |
Steps defines a series of sequential/parallel data flow steps The first dimension represents parallel data flows. The second sequential components within the same parallel data flow. |
true |
triggers | []enum |
Triggers |
true |
Plotter.spec.flows[index].subFlows[index].steps[index][index]
DataFlowStep contains details on a single data flow step
Name | Type | Description | Required |
---|---|---|---|
cluster | string |
Name of the cluster this step is executed on |
true |
name | string |
Name of the step |
true |
template | string |
Template is the name of the template to execute the step The full details of the template can be extracted from Plotter.spec.templates list field. |
true |
parameters | object |
Step parameters TODO why not flatten the parameters into this data flow step |
false |
Plotter.spec.flows[index].subFlows[index].steps[index][index].parameters
Step parameters TODO why not flatten the parameters into this data flow step
Plotter.spec.flows[index].subFlows[index].steps[index][index].parameters.action[index]
Name | Type | Description | Required |
---|---|---|---|
name | string |
|
true |
Plotter.spec.flows[index].subFlows[index].steps[index][index].parameters.api
ResourceDetails includes asset connection details
Name | Type | Description | Required |
---|---|---|---|
connection | object |
Connection information |
true |
dataFormat | string |
Data format |
false |
Plotter.spec.flows[index].subFlows[index].steps[index][index].parameters.api.connection
Connection information
Name | Type | Description | Required |
---|---|---|---|
name | string |
Name of the connection to the data source |
true |
Plotter.spec.flows[index].subFlows[index].steps[index][index].parameters.args[index]
StepArgument describes a step: it could be assetID or an endpoint of another step
Name | Type | Description | Required |
---|---|---|---|
api | object |
API holds information for accessing a module instance |
false |
assetId | string |
AssetID identifies the source asset of this step |
false |
Plotter.spec.flows[index].subFlows[index].steps[index][index].parameters.args[index].api
API holds information for accessing a module instance
Name | Type | Description | Required |
---|---|---|---|
connection | object |
Connection information |
true |
dataFormat | string |
Data format |
false |
Plotter.spec.flows[index].subFlows[index].steps[index][index].parameters.args[index].api.connection
Connection information
Name | Type | Description | Required |
---|---|---|---|
name | string |
Name of the connection to the data source |
true |
Plotter.spec.templates[key]
Template contains basic information about the required modules to serve the fybrikapplication e.g., the module helm chart name.
Name | Type | Description | Required |
---|---|---|---|
modules | []object |
Modules is a list of dependent modules. e.g., if a plugin module is used then the service module is used in should appear first in the modules list of the same template. If the modules list contains more than one module, the first module in the list is referred to as the "primary module" of which all the parameters to this template are sent to. |
true |
name | string |
Name of the template |
false |
Plotter.spec.templates[key].modules[index]
ModuleInfo is a copy of FybrikModule Custom Resource. It contains information to instantiate resource of type FybrikModule.
Name | Type | Description | Required |
---|---|---|---|
capability | string |
Module capability |
true |
chart | object |
Chart contains the information needed to use helm to install the capability |
true |
name | string |
Name of the module |
true |
type | string |
May be one of service, config or plugin Service: Means that the control plane deploys the component that performs the capability Config: Another pre-installed service performs the capability and the module deployed configures it for the particular workload or dataset Plugin: Indicates that this module performs a capability as part of another service or module rather than as a stand-alone module |
true |
scope | enum |
Scope indicates at what level the capability is used: workload, asset, cluster If not indicated it is assumed to be asset Enum: asset, workload, cluster |
false |
Plotter.spec.templates[key].modules[index].chart
Chart contains the information needed to use helm to install the capability
Name | Type | Description | Required |
---|---|---|---|
name | string |
Name of helm chart |
true |
chartPullSecret | string |
Name of secret containing helm registry credentials |
false |
values | map[string]string |
Values to pass to helm chart installation |
false |
Plotter.spec.appSelector
Selector enables to connect the resource to the application Application labels should match the labels in the selector. For some flows the selector may not be used.
Name | Type | Description | Required |
---|---|---|---|
workloadSelector | object |
WorkloadSelector enables to connect the resource to the application Application labels should match the labels in the selector. |
true |
clusterName | string |
Cluster name |
false |
Plotter.spec.appSelector.workloadSelector
WorkloadSelector enables to connect the resource to the application Application labels should match the labels in the selector.
Name | Type | Description | Required |
---|---|---|---|
matchExpressions | []object |
matchExpressions is a list of label selector requirements. The requirements are ANDed. |
false |
matchLabels | map[string]string |
matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. |
false |
Plotter.spec.appSelector.workloadSelector.matchExpressions[index]
A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
Name | Type | Description | Required |
---|---|---|---|
key | string |
key is the label key that the selector applies to. |
true |
operator | string |
operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. |
true |
values | []string |
values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. |
false |
Plotter.status
PlotterStatus defines the observed state of Plotter This includes readiness, error message, and indicators received from blueprint resources owned by the Plotter for cleanup and status monitoring
Name | Type | Description | Required |
---|---|---|---|
assets | map[string]object |
Assets is a map containing the status per asset. The key of this map is assetId |
false |
blueprints | map[string]object |
|
false |
conditions | []object |
Conditions represent the possible error and failure conditions |
false |
flows | map[string]object |
Flows is a map containing the status for each flow the key is the flow name |
false |
observedGeneration | integer |
ObservedGeneration is taken from the Plotter metadata. This is used to determine during reconcile whether reconcile was called because the desired state changed, or whether status of the allocated blueprints should be checked. Format: int64 |
false |
observedState | object |
ObservedState includes information to be reported back to the FybrikApplication resource It includes readiness and error indications, as well as user instructions |
false |
readyTimestamp | string |
Format: date-time |
false |
Plotter.status.assets[key]
ObservedState represents a part of the generated Blueprint/Plotter resource status that allows update of FybrikApplication status
Name | Type | Description | Required |
---|---|---|---|
error | string |
Error indicates that there has been an error to orchestrate the modules and provides the error message |
false |
ready | boolean |
Ready represents that the modules have been orchestrated successfully and the data is ready for usage |
false |
Plotter.status.blueprints[key]
MetaBlueprint defines blueprint metadata (name, namespace) and status
Name | Type | Description | Required |
---|---|---|---|
name | string |
|
true |
namespace | string |
|
true |
status | object |
BlueprintStatus defines the observed state of Blueprint This includes readiness, error message, and indicators for the Kubernetes resources owned by the Blueprint for cleanup and status monitoring |
true |
Plotter.status.blueprints[key].status
BlueprintStatus defines the observed state of Blueprint This includes readiness, error message, and indicators for the Kubernetes resources owned by the Blueprint for cleanup and status monitoring
Name | Type | Description | Required |
---|---|---|---|
modules | map[string]object |
ModulesState is a map which holds the status of each module its key is the moduleInstanceName which is the unique name for the deployed instance related to this workload |
false |
observedGeneration | integer |
ObservedGeneration is taken from the Blueprint metadata. This is used to determine during reconcile whether reconcile was called because the desired state changed, or whether status of the allocated resources should be checked. Format: int64 |
false |
observedState | object |
ObservedState includes information to be reported back to the FybrikApplication resource It includes readiness and error indications, as well as user instructions |
false |
releases | map[string]integer |
Releases map each release to the observed generation of the blueprint containing this release. At the end of reconcile, each release should be mapped to the latest blueprint version or be uninstalled. |
false |
Plotter.status.blueprints[key].status.modules[key]
ObservedState represents a part of the generated Blueprint/Plotter resource status that allows update of FybrikApplication status
Name | Type | Description | Required |
---|---|---|---|
error | string |
Error indicates that there has been an error to orchestrate the modules and provides the error message |
false |
ready | boolean |
Ready represents that the modules have been orchestrated successfully and the data is ready for usage |
false |
Plotter.status.blueprints[key].status.observedState
ObservedState includes information to be reported back to the FybrikApplication resource It includes readiness and error indications, as well as user instructions
Name | Type | Description | Required |
---|---|---|---|
error | string |
Error indicates that there has been an error to orchestrate the modules and provides the error message |
false |
ready | boolean |
Ready represents that the modules have been orchestrated successfully and the data is ready for usage |
false |
Plotter.status.conditions[index]
Condition describes the state of a FybrikApplication at a certain point.
Name | Type | Description | Required |
---|---|---|---|
type | string |
Type of the condition |
true |
message | string |
Message contains the details of the current condition |
false |
observedGeneration | integer |
ObservedGeneration is the version of the resource for which the condition has been evaluated Format: int64 |
false |
status | enum |
Status of the condition, one of (`True`, `False`, `Unknown`). Enum: True, False, Unknown Default: Unknown |
false |
Plotter.status.flows[key]
FlowStatus includes information to be reported back to the FybrikApplication resource It holds the status per data flow
Plotter.status.flows[key].subFlows[key]
ObservedState represents a part of the generated Blueprint/Plotter resource status that allows update of FybrikApplication status
Name | Type | Description | Required |
---|---|---|---|
error | string |
Error indicates that there has been an error to orchestrate the modules and provides the error message |
false |
ready | boolean |
Ready represents that the modules have been orchestrated successfully and the data is ready for usage |
false |
Plotter.status.flows[key].status
ObservedState includes information about the current flow It includes readiness and error indications, as well as user instructions
Name | Type | Description | Required |
---|---|---|---|
error | string |
Error indicates that there has been an error to orchestrate the modules and provides the error message |
false |
ready | boolean |
Ready represents that the modules have been orchestrated successfully and the data is ready for usage |
false |
Plotter.status.observedState
ObservedState includes information to be reported back to the FybrikApplication resource It includes readiness and error indications, as well as user instructions
Name | Type | Description | Required |
---|---|---|---|
error | string |
Error indicates that there has been an error to orchestrate the modules and provides the error message |
false |
ready | boolean |
Ready represents that the modules have been orchestrated successfully and the data is ready for usage |
false |
katalog.fybrik.io/v1alpha1
Resource Types:
Asset
Asset defines an asset in the catalog
Asset.spec
Asset.spec.details
Asset details
Name | Type | Description | Required |
---|---|---|---|
connection | object |
Connection information |
true |
dataFormat | string |
Data format |
false |
Asset.spec.details.connection
Connection information
Name | Type | Description | Required |
---|---|---|---|
name | string |
Name of the connection to the data source |
true |
Asset.spec.metadata
Asset metadata
Name | Type | Description | Required |
---|---|---|---|
columns | []object |
Columns associated with the asset |
false |
geography | string |
Geography of the resource |
false |
name | string |
Name of the resource |
false |
owner | string |
Owner of the resource |
false |
tags | object |
Tags associated with the asset |
false |
Asset.spec.metadata.columns[index]
ResourceColumn represents a column in a tabular resource
Name | Type | Description | Required |
---|---|---|---|
name | string |
Name of the column |
true |
tags | object |
Tags associated with the column |
false |
Asset.spec.secretRef
Reference to a Secret resource holding credentials for this asset
Name | Type | Description | Required |
---|---|---|---|
name | string |
Name of the Secret resource |
true |
namespace | string |
Namespace of the Secret resource. If it is empty then the asset namespace is used. |
false |