API Reference
Packages:
app.fybrik.io/v1alpha1
Resource Types:
Blueprint
Blueprint is the Schema for the blueprints API
Name | Type | Description | Required |
---|---|---|---|
apiVersion | string | app.fybrik.io/v1alpha1 | true |
kind | string | Blueprint | true |
metadata | object | Refer to the Kubernetes API documentation for the fields of the `metadata` field. | true |
spec | object |
BlueprintSpec defines the desired state of Blueprint, which defines the components of the workload's data path that run in a particular cluster. In a single cluster environment there is one blueprint per workload (FybrikApplication). In a multi-cluster environment there is one Blueprint per cluster per workload (FybrikApplication). |
false |
status | object |
BlueprintStatus defines the observed state of Blueprint This includes readiness, error message, and indicators for the Kubernetes resources owned by the Blueprint for cleanup and status monitoring |
false |
Blueprint.spec
BlueprintSpec defines the desired state of Blueprint, which defines the components of the workload's data path that run in a particular cluster. In a single cluster environment there is one blueprint per workload (FybrikApplication). In a multi-cluster environment there is one Blueprint per cluster per workload (FybrikApplication).
Name | Type | Description | Required |
---|---|---|---|
cluster | string |
Cluster indicates the cluster on which the Blueprint runs |
true |
modules | map[string]object |
Modules is a map which contains modules that indicate the data path components that run in this cluster The map key is moduleInstanceName which is the unique name for the deployed instance related to this workload |
true |
modulesNamespace | string |
ModulesNamespace is the namespace where modules should be allocated |
true |
Blueprint.spec.modules[key]
BlueprintModule is a copy of a FybrikModule Custom Resource. It contains the information necessary to instantiate a datapath component, including the parameters relevant for the particular workload.
Name | Type | Description | Required |
---|---|---|---|
arguments | object |
Arguments are the input parameters for a specific instance of a module. |
false |
assetIds | []string |
assetIDs indicate the assets processed by this module. Included so we can track asset status as well as module status in the future. |
false |
chart | object |
Chart contains the location of the helm chart with info detailing how to deploy |
true |
name | string |
Name of the FybrikModule on which this is based |
true |
Blueprint.spec.modules[key].arguments
Arguments are the input parameters for a specific instance of a module.
Name | Type | Description | Required |
---|---|---|---|
appSelector | object |
Application selector is used to identify the user workload. It is obtained from FybrikApplication spec. |
false |
copy | object |
CopyArgs are parameters specific to modules that copy data from one data store to another. |
false |
labels | map[string]string |
Labels of FybrikApplication |
false |
read | []object |
ReadArgs are parameters that are specific to modules that enable an application to read data |
false |
write | []object |
WriteArgs are parameters that are specific to modules that enable an application to write data |
false |
Blueprint.spec.modules[key].arguments.appSelector
Application selector is used to identify the user workload. It is obtained from FybrikApplication spec.
Name | Type | Description | Required |
---|---|---|---|
matchExpressions | []object |
matchExpressions is a list of label selector requirements. The requirements are ANDed. |
false |
matchLabels | map[string]string |
matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. |
false |
Blueprint.spec.modules[key].arguments.appSelector.matchExpressions[index]
A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
Name | Type | Description | Required |
---|---|---|---|
values | []string |
values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. |
false |
key | string |
key is the label key that the selector applies to. |
true |
operator | string |
operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. |
true |
Blueprint.spec.modules[key].arguments.copy
CopyArgs are parameters specific to modules that copy data from one data store to another.
Name | Type | Description | Required |
---|---|---|---|
transformations | []object |
Transformations are different types of processing that may be done to the data as it is copied. |
false |
assetID | string |
AssetID identifies the asset to be used for accessing the data when it is ready It is copied from the FybrikApplication resource |
true |
destination | object |
Destination is the data store to which the data will be copied |
true |
source | object |
Source is the where the data currently resides |
true |
Blueprint.spec.modules[key].arguments.copy.transformations[index]
Name | Type | Description | Required |
---|---|---|---|
name | string |
|
true |
Blueprint.spec.modules[key].arguments.copy.destination
Destination is the data store to which the data will be copied
Name | Type | Description | Required |
---|---|---|---|
format | string |
Format represents data format (e.g. parquet) as received from catalog connectors |
false |
connection | object |
Connection has the relevant details for accesing the data (url, table, ssl, etc.) |
true |
vault | map[string]object |
Holds details for retrieving credentials by the modules from Vault store. It is a map so that different credentials can be stored for the different DataFlow operations. |
true |
Blueprint.spec.modules[key].arguments.copy.destination.connection
Connection has the relevant details for accesing the data (url, table, ssl, etc.)
Name | Type | Description | Required |
---|---|---|---|
name | string |
|
true |
Blueprint.spec.modules[key].arguments.copy.destination.vault[key]
Holds details for retrieving credentials from Vault store.
Name | Type | Description | Required |
---|---|---|---|
address | string |
Address is Vault address |
true |
authPath | string |
AuthPath is the path to auth method i.e. kubernetes |
true |
role | string |
Role is the Vault role used for retrieving the credentials |
true |
secretPath | string |
SecretPath is the path of the secret holding the Credentials in Vault |
true |
Blueprint.spec.modules[key].arguments.copy.source
Source is the where the data currently resides
Name | Type | Description | Required |
---|---|---|---|
format | string |
Format represents data format (e.g. parquet) as received from catalog connectors |
false |
connection | object |
Connection has the relevant details for accesing the data (url, table, ssl, etc.) |
true |
vault | map[string]object |
Holds details for retrieving credentials by the modules from Vault store. It is a map so that different credentials can be stored for the different DataFlow operations. |
true |
Blueprint.spec.modules[key].arguments.copy.source.connection
Connection has the relevant details for accesing the data (url, table, ssl, etc.)
Name | Type | Description | Required |
---|---|---|---|
name | string |
|
true |
Blueprint.spec.modules[key].arguments.copy.source.vault[key]
Holds details for retrieving credentials from Vault store.
Name | Type | Description | Required |
---|---|---|---|
address | string |
Address is Vault address |
true |
authPath | string |
AuthPath is the path to auth method i.e. kubernetes |
true |
role | string |
Role is the Vault role used for retrieving the credentials |
true |
secretPath | string |
SecretPath is the path of the secret holding the Credentials in Vault |
true |
Blueprint.spec.modules[key].arguments.read[index]
ReadModuleArgs define the input parameters for modules that read data from location A
Name | Type | Description | Required |
---|---|---|---|
transformations | []object |
Transformations are different types of processing that may be done to the data |
false |
assetID | string |
AssetID identifies the asset to be used for accessing the data when it is ready It is copied from the FybrikApplication resource |
true |
source | object |
Source of the read path module |
true |
Blueprint.spec.modules[key].arguments.read[index].transformations[index]
Name | Type | Description | Required |
---|---|---|---|
name | string |
|
true |
Blueprint.spec.modules[key].arguments.read[index].source
Source of the read path module
Name | Type | Description | Required |
---|---|---|---|
format | string |
Format represents data format (e.g. parquet) as received from catalog connectors |
false |
connection | object |
Connection has the relevant details for accesing the data (url, table, ssl, etc.) |
true |
vault | map[string]object |
Holds details for retrieving credentials by the modules from Vault store. It is a map so that different credentials can be stored for the different DataFlow operations. |
true |
Blueprint.spec.modules[key].arguments.read[index].source.connection
Connection has the relevant details for accesing the data (url, table, ssl, etc.)
Name | Type | Description | Required |
---|---|---|---|
name | string |
|
true |
Blueprint.spec.modules[key].arguments.read[index].source.vault[key]
Holds details for retrieving credentials from Vault store.
Name | Type | Description | Required |
---|---|---|---|
address | string |
Address is Vault address |
true |
authPath | string |
AuthPath is the path to auth method i.e. kubernetes |
true |
role | string |
Role is the Vault role used for retrieving the credentials |
true |
secretPath | string |
SecretPath is the path of the secret holding the Credentials in Vault |
true |
Blueprint.spec.modules[key].arguments.write[index]
WriteModuleArgs define the input parameters for modules that write data to location B
Name | Type | Description | Required |
---|---|---|---|
transformations | []object |
Transformations are different types of processing that may be done to the data as it is written. |
false |
assetID | string |
AssetID identifies the asset to be used for accessing the data when it is ready It is copied from the FybrikApplication resource |
true |
destination | object |
Destination is the data store to which the data will be written |
true |
Blueprint.spec.modules[key].arguments.write[index].transformations[index]
Name | Type | Description | Required |
---|---|---|---|
name | string |
|
true |
Blueprint.spec.modules[key].arguments.write[index].destination
Destination is the data store to which the data will be written
Name | Type | Description | Required |
---|---|---|---|
format | string |
Format represents data format (e.g. parquet) as received from catalog connectors |
false |
connection | object |
Connection has the relevant details for accesing the data (url, table, ssl, etc.) |
true |
vault | map[string]object |
Holds details for retrieving credentials by the modules from Vault store. It is a map so that different credentials can be stored for the different DataFlow operations. |
true |
Blueprint.spec.modules[key].arguments.write[index].destination.connection
Connection has the relevant details for accesing the data (url, table, ssl, etc.)
Name | Type | Description | Required |
---|---|---|---|
name | string |
|
true |
Blueprint.spec.modules[key].arguments.write[index].destination.vault[key]
Holds details for retrieving credentials from Vault store.
Name | Type | Description | Required |
---|---|---|---|
address | string |
Address is Vault address |
true |
authPath | string |
AuthPath is the path to auth method i.e. kubernetes |
true |
role | string |
Role is the Vault role used for retrieving the credentials |
true |
secretPath | string |
SecretPath is the path of the secret holding the Credentials in Vault |
true |
Blueprint.spec.modules[key].chart
Chart contains the location of the helm chart with info detailing how to deploy
Name | Type | Description | Required |
---|---|---|---|
chartPullSecret | string |
Name of secret containing helm registry credentials |
false |
values | map[string]string |
Values to pass to helm chart installation |
false |
name | string |
Name of helm chart |
true |
Blueprint.status
BlueprintStatus defines the observed state of Blueprint This includes readiness, error message, and indicators for the Kubernetes resources owned by the Blueprint for cleanup and status monitoring
Name | Type | Description | Required |
---|---|---|---|
modules | map[string]object |
ModulesState is a map which holds the status of each module its key is the moduleInstanceName which is the unique name for the deployed instance related to this workload |
false |
observedGeneration | integer |
ObservedGeneration is taken from the Blueprint metadata. This is used to determine during reconcile whether reconcile was called because the desired state changed, or whether status of the allocated resources should be checked. Format: int64 |
false |
observedState | object |
ObservedState includes information to be reported back to the FybrikApplication resource It includes readiness and error indications, as well as user instructions |
false |
releases | map[string]integer |
Releases map each release to the observed generation of the blueprint containing this release. At the end of reconcile, each release should be mapped to the latest blueprint version or be uninstalled. |
false |
Blueprint.status.modules[key]
ObservedState represents a part of the generated Blueprint/Plotter resource status that allows update of FybrikApplication status
Name | Type | Description | Required |
---|---|---|---|
error | string |
Error indicates that there has been an error to orchestrate the modules and provides the error message |
false |
ready | boolean |
Ready represents that the modules have been orchestrated successfully and the data is ready for usage |
false |
Blueprint.status.observedState
ObservedState includes information to be reported back to the FybrikApplication resource It includes readiness and error indications, as well as user instructions
Name | Type | Description | Required |
---|---|---|---|
error | string |
Error indicates that there has been an error to orchestrate the modules and provides the error message |
false |
ready | boolean |
Ready represents that the modules have been orchestrated successfully and the data is ready for usage |
false |
FybrikApplication
FybrikApplication provides information about the application whose data is being operated on, the nature of the processing, and the data sets chosen for processing by the application. The FybrikApplication controller obtains instructions regarding any governance related changes that must be performed on the data, identifies the modules capable of performing such changes, and finally generates the Plotter which defines the secure runtime environment and all the components in it. This runtime environment provides the application with access to the data requested in a secure manner and without having to provide any credentials for the data sets. The credentials are obtained automatically by the manager from the credential management system.
Name | Type | Description | Required |
---|---|---|---|
apiVersion | string | app.fybrik.io/v1alpha1 | true |
kind | string | FybrikApplication | true |
metadata | object | Refer to the Kubernetes API documentation for the fields of the `metadata` field. | true |
spec | object |
FybrikApplicationSpec defines data flows needed by the application, the purpose and other contextual information about the application. Read flow - if selector is populated, fybrik builds a data plane for reading the specified data sets Ingest flow - if no selector, and data/copy/required is true then the data specified is copied into a bucket allocated by fybrik and is cataloged in the data catalog |
false |
status | object |
FybrikApplicationStatus defines the observed state of FybrikApplication. |
false |
FybrikApplication.spec
FybrikApplicationSpec defines data flows needed by the application, the purpose and other contextual information about the application. Read flow - if selector is populated, fybrik builds a data plane for reading the specified data sets Ingest flow - if no selector, and data/copy/required is true then the data specified is copied into a bucket allocated by fybrik and is cataloged in the data catalog
Name | Type | Description | Required |
---|---|---|---|
secretRef | string |
SecretRef points to the secret that holds credentials for each system the user has been authenticated with. The secret is deployed in FybrikApplication namespace. |
false |
selector | object |
Selector enables to connect the resource to the application Application labels should match the labels in the selector. |
false |
appInfo | object |
AppInfo contains information describing the reasons for the processing that will be done by the application. |
true |
data | []object |
Data contains the identifiers of the data to be used by the Data Scientist's application, and the protocol used to access it and the format expected. |
true |
FybrikApplication.spec.selector
Selector enables to connect the resource to the application Application labels should match the labels in the selector.
Name | Type | Description | Required |
---|---|---|---|
clusterName | string |
Cluster name |
false |
workloadSelector | object |
WorkloadSelector enables to connect the resource to the application Application labels should match the labels in the selector. |
true |
FybrikApplication.spec.selector.workloadSelector
WorkloadSelector enables to connect the resource to the application Application labels should match the labels in the selector.
Name | Type | Description | Required |
---|---|---|---|
matchExpressions | []object |
matchExpressions is a list of label selector requirements. The requirements are ANDed. |
false |
matchLabels | map[string]string |
matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. |
false |
FybrikApplication.spec.selector.workloadSelector.matchExpressions[index]
A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
Name | Type | Description | Required |
---|---|---|---|
values | []string |
values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. |
false |
key | string |
key is the label key that the selector applies to. |
true |
operator | string |
operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. |
true |
FybrikApplication.spec.data[index]
DataContext indicates data set chosen by the Data Scientist to be used by his application, and includes information about the data format and technologies used by the application to access the data.
Name | Type | Description | Required |
---|---|---|---|
catalogService | string |
CatalogService represents the catalog service for accessing the requested dataset. If not specified, the enterprise catalog service will be used. |
false |
dataSetID | string |
DataSetID is a unique identifier of the dataset chosen from the data catalog for processing by the data user application. |
true |
requirements | object |
Requirements from the system |
true |
FybrikApplication.spec.data[index].requirements
Requirements from the system
Name | Type | Description | Required |
---|---|---|---|
copy | object |
CopyRequrements include the requirements for copying the data |
false |
interface | object |
Interface indicates the protocol and format expected by the data user |
true |
FybrikApplication.spec.data[index].requirements.copy
CopyRequrements include the requirements for copying the data
Name | Type | Description | Required |
---|---|---|---|
catalog | object |
Catalog indicates that the data asset must be cataloged. |
false |
required | boolean |
Required indicates that the data must be copied. |
false |
FybrikApplication.spec.data[index].requirements.copy.catalog
Catalog indicates that the data asset must be cataloged.
Name | Type | Description | Required |
---|---|---|---|
catalogID | string |
CatalogID specifies the catalog where the data will be cataloged. |
false |
service | string |
CatalogService specifies the datacatalog service that will be used for catalogging the data into. |
false |
FybrikApplication.spec.data[index].requirements.interface
Interface indicates the protocol and format expected by the data user
Name | Type | Description | Required |
---|---|---|---|
dataformat | string |
DataFormat defines the data format type |
false |
protocol | string |
Protocol defines the interface protocol used for data transactions |
true |
FybrikApplication.status
FybrikApplicationStatus defines the observed state of FybrikApplication.
Name | Type | Description | Required |
---|---|---|---|
assetStates | map[string]object |
AssetStates provides a status per asset |
false |
errorMessage | string |
ErrorMessage indicates that an error has happened during the reconcile, unrelated to a specific asset |
false |
generated | object |
Generated resource identifier |
false |
observedGeneration | integer |
ObservedGeneration is taken from the FybrikApplication metadata. This is used to determine during reconcile whether reconcile was called because the desired state changed, or whether the Blueprint status changed. Format: int64 |
false |
provisionedStorage | map[string]object |
ProvisionedStorage maps a dataset (identified by AssetID) to the new provisioned bucket. It allows FybrikApplication controller to manage buckets in case the spec has been modified, an error has occurred, or a delete event has been received. ProvisionedStorage has the information required to register the dataset once the owned plotter resource is ready |
false |
ready | boolean |
Ready is true if all specified assets are either ready to be used or are denied access. |
false |
validApplication | string |
ValidApplication indicates whether the FybrikApplication is valid given the defined taxonomy |
false |
validatedGeneration | integer |
ValidatedGeneration is the version of the FyrbikApplication that has been validated with the taxonomy defined. Format: int64 |
false |
FybrikApplication.status.assetStates[key]
AssetState defines the observed state of an asset
Name | Type | Description | Required |
---|---|---|---|
catalogedAsset | string |
CatalogedAsset provides a new asset identifier after being registered in the enterprise catalog |
false |
conditions | []object |
Conditions indicate the asset state (Ready, Deny, Error) |
false |
endpoint | object |
Endpoint provides the endpoint spec from which the asset will be served to the application |
false |
FybrikApplication.status.assetStates[key].conditions[index]
Condition describes the state of a FybrikApplication at a certain point.
Name | Type | Description | Required |
---|---|---|---|
message | string |
Message contains the details of the current condition |
false |
observedGeneration | integer |
ObservedGeneration is the version of the resource for which the condition has been evaluated Format: int64 |
false |
status | enum |
Status of the condition, one of (`True`, `False`, `Unknown`). Enum: True, False, Unknown Default: Unknown |
false |
type | string |
Type of the condition |
true |
FybrikApplication.status.assetStates[key].endpoint
Endpoint provides the endpoint spec from which the asset will be served to the application
Name | Type | Description | Required |
---|---|---|---|
name | string |
|
true |
FybrikApplication.status.generated
Generated resource identifier
Name | Type | Description | Required |
---|---|---|---|
appVersion | integer |
Version of FybrikApplication that has generated this resource Format: int64 |
true |
kind | string |
Kind of the resource (Blueprint, Plotter) |
true |
name | string |
Name of the resource |
true |
namespace | string |
Namespace of the resource |
true |
FybrikApplication.status.provisionedStorage[key]
DatasetDetails holds details of the provisioned storage
Name | Type | Description | Required |
---|---|---|---|
datasetRef | string |
Reference to a Dataset resource containing the request to provision storage |
false |
secretRef | string |
Reference to a secret where the credentials are stored |
false |
FybrikModule
FybrikModule is a description of an injectable component. the parameters it requires, as well as the specification of how to instantiate such a component. It is used as metadata only. There is no status nor reconciliation.
Name | Type | Description | Required |
---|---|---|---|
apiVersion | string | app.fybrik.io/v1alpha1 | true |
kind | string | FybrikModule | true |
metadata | object | Refer to the Kubernetes API documentation for the fields of the `metadata` field. | true |
status | object |
FybrikModuleStatus defines the observed state of FybrikModule. |
false |
spec | object |
FybrikModuleSpec contains the info common to all modules, which are one of the components that process, load, write, audit, monitor the data used by the data scientist's application. |
true |
FybrikModule.status
FybrikModuleStatus defines the observed state of FybrikModule.
Name | Type | Description | Required |
---|---|---|---|
conditions | []object |
Conditions indicate the module states with respect to validation |
false |
FybrikModule.status.conditions[index]
Condition describes the state of a FybrikApplication at a certain point.
Name | Type | Description | Required |
---|---|---|---|
message | string |
Message contains the details of the current condition |
false |
observedGeneration | integer |
ObservedGeneration is the version of the resource for which the condition has been evaluated Format: int64 |
false |
status | enum |
Status of the condition, one of (`True`, `False`, `Unknown`). Enum: True, False, Unknown Default: Unknown |
false |
type | string |
Type of the condition |
true |
FybrikModule.spec
FybrikModuleSpec contains the info common to all modules, which are one of the components that process, load, write, audit, monitor the data used by the data scientist's application.
Name | Type | Description | Required |
---|---|---|---|
dependencies | []object |
Other components that must be installed in order for this module to work |
false |
description | string |
An explanation of what this module does |
false |
pluginType | string |
Plugin type indicates the plugin technology used to invoke the capabilities Ex: vault, fybrik-wasm... Should be provided if type is plugin |
false |
statusIndicators | []object |
StatusIndicators allow to check status of a non-standard resource that can not be computed by helm/kstatus |
false |
capabilities | []object |
Capabilities declares what this module knows how to do and the types of data it knows how to handle The key to the map is a CapabilityType string |
true |
chart | object |
Reference to a Helm chart that allows deployment of the resources required for this module |
true |
type | string |
May be one of service, config or plugin Service: Means that the control plane deploys the component that performs the capability Config: Another pre-installed service performs the capability and the module deployed configures it for the particular workload or dataset Plugin: Indicates that this module performs a capability as part of another service or module rather than as a stand-alone module |
true |
FybrikModule.spec.dependencies[index]
Dependency details another component on which this module relies - i.e. a pre-requisit
Name | Type | Description | Required |
---|---|---|---|
name | string |
Name is the name of the dependent component |
true |
type | enum |
Type provides information used in determining how to instantiate the component Enum: module, connector, feature |
true |
FybrikModule.spec.statusIndicators[index]
ResourceStatusIndicator is used to determine the status of an orchestrated resource
Name | Type | Description | Required |
---|---|---|---|
errorMessage | string |
ErrorMessage specifies the resource field to check for an error, e.g. status.errorMsg |
false |
failureCondition | string |
FailureCondition specifies a condition that indicates the resource failure It uses kubernetes label selection syntax (https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/) |
false |
kind | string |
Kind provides information about the resource kind |
true |
successCondition | string |
SuccessCondition specifies a condition that indicates that the resource is ready It uses kubernetes label selection syntax (https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/) |
true |
FybrikModule.spec.capabilities[index]
Capability declares what this module knows how to do and the types of data it knows how to handle
Name | Type | Description | Required |
---|---|---|---|
actions | []object |
Actions are the data transformations that the module supports |
false |
api | object |
API indicates to the application how to access the capabilities provided by the module |
false |
plugins | []object |
Plugins enable the module to add libraries to perform actions rather than implementing them by itself |
false |
scope | enum |
Scope indicates at what level the capability is used: workload, asset, cluster If not indicated it is assumed to be asset Enum: asset, workload, cluster |
false |
supportedInterfaces | []object |
Copy should have one or more instances in the list, and its content should have source and sink Read should have one or more instances in the list, each with source populated Write should have one or more instances in the list, each with sink populated This field may not be required if not handling data |
false |
capability | string |
Capability declares what this module knows how to do - ex: read, write, transform... |
true |
FybrikModule.spec.capabilities[index].actions[index]
Name | Type | Description | Required |
---|---|---|---|
name | string |
Unique name of an action supported by the module |
true |
FybrikModule.spec.capabilities[index].api
API indicates to the application how to access the capabilities provided by the module
Name | Type | Description | Required |
---|---|---|---|
dataFormat | string |
Data format |
false |
connection | object |
Connection information |
true |
FybrikModule.spec.capabilities[index].api.connection
Connection information
Name | Type | Description | Required |
---|---|---|---|
name | string |
|
true |
FybrikModule.spec.capabilities[index].plugins[index]
Name | Type | Description | Required |
---|---|---|---|
dataFormat | string |
DataFormat indicates the format of data the plugin knows how to process |
true |
pluginType | string |
PluginType indicates the technology used for the module and the plugin to interact The values supported should come from the module taxonomy Examples of such mechanisms are vault plugins, wasm, etc |
true |
FybrikModule.spec.capabilities[index].supportedInterfaces[index]
ModuleInOut specifies the protocol and format of the data input and output by the module - if any
Name | Type | Description | Required |
---|---|---|---|
sink | object |
Sink specifies the output data protocol and format |
false |
source | object |
Source specifies the input data protocol and format |
false |
FybrikModule.spec.capabilities[index].supportedInterfaces[index].sink
Sink specifies the output data protocol and format
Name | Type | Description | Required |
---|---|---|---|
dataformat | string |
DataFormat defines the data format type |
false |
protocol | string |
Protocol defines the interface protocol used for data transactions |
true |
FybrikModule.spec.capabilities[index].supportedInterfaces[index].source
Source specifies the input data protocol and format
Name | Type | Description | Required |
---|---|---|---|
dataformat | string |
DataFormat defines the data format type |
false |
protocol | string |
Protocol defines the interface protocol used for data transactions |
true |
FybrikModule.spec.chart
Reference to a Helm chart that allows deployment of the resources required for this module
Name | Type | Description | Required |
---|---|---|---|
chartPullSecret | string |
Name of secret containing helm registry credentials |
false |
values | map[string]string |
Values to pass to helm chart installation |
false |
name | string |
Name of helm chart |
true |
FybrikStorageAccount
FybrikStorageAccount defines a storage account used for copying data. Only S3 based storage is supported. It contains endpoint, region and a reference to the credentials a Owner of the asset is responsible to store the credentials
Name | Type | Description | Required |
---|---|---|---|
apiVersion | string | app.fybrik.io/v1alpha1 | true |
kind | string | FybrikStorageAccount | true |
metadata | object | Refer to the Kubernetes API documentation for the fields of the `metadata` field. | true |
spec | object |
FybrikStorageAccountSpec defines the desired state of FybrikStorageAccount |
false |
status | object |
FybrikStorageAccountStatus defines the observed state of FybrikStorageAccount |
false |
FybrikStorageAccount.spec
FybrikStorageAccountSpec defines the desired state of FybrikStorageAccount
Name | Type | Description | Required |
---|---|---|---|
endpoints | map[string]string |
Endpoints based on regions |
true |
id | string |
Identification of a storage account |
true |
secretRef | string |
A name of k8s secret deployed in the control plane. This secret includes secretKey and accessKey credentials for S3 bucket |
true |
Plotter
Plotter is the Schema for the plotters API
Name | Type | Description | Required |
---|---|---|---|
apiVersion | string | app.fybrik.io/v1alpha1 | true |
kind | string | Plotter | true |
metadata | object | Refer to the Kubernetes API documentation for the fields of the `metadata` field. | true |
spec | object |
PlotterSpec defines the desired state of Plotter, which is applied in a multi-clustered environment. Plotter declares what needs to be installed and where (as blueprints running on remote clusters) which provides the Data Scientist's application with secure and governed access to the data requested in the FybrikApplication. |
false |
status | object |
PlotterStatus defines the observed state of Plotter This includes readiness, error message, and indicators received from blueprint resources owned by the Plotter for cleanup and status monitoring |
false |
Plotter.spec
PlotterSpec defines the desired state of Plotter, which is applied in a multi-clustered environment. Plotter declares what needs to be installed and where (as blueprints running on remote clusters) which provides the Data Scientist's application with secure and governed access to the data requested in the FybrikApplication.
Name | Type | Description | Required |
---|---|---|---|
appSelector | object |
Selector enables to connect the resource to the application Application labels should match the labels in the selector. For some flows the selector may not be used. |
false |
assets | map[string]object |
Assets is a map holding information about the assets The key is the assetID |
true |
flows | []object |
|
true |
modulesNamespace | string |
ModulesNamespace is the namespace where modules should be allocated |
true |
templates | map[string]object |
Templates is a map holding the templates used in this plotter steps The key is the template name |
true |
Plotter.spec.appSelector
Selector enables to connect the resource to the application Application labels should match the labels in the selector. For some flows the selector may not be used.
Name | Type | Description | Required |
---|---|---|---|
clusterName | string |
Cluster name |
false |
workloadSelector | object |
WorkloadSelector enables to connect the resource to the application Application labels should match the labels in the selector. |
true |
Plotter.spec.appSelector.workloadSelector
WorkloadSelector enables to connect the resource to the application Application labels should match the labels in the selector.
Name | Type | Description | Required |
---|---|---|---|
matchExpressions | []object |
matchExpressions is a list of label selector requirements. The requirements are ANDed. |
false |
matchLabels | map[string]string |
matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. |
false |
Plotter.spec.appSelector.workloadSelector.matchExpressions[index]
A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
Name | Type | Description | Required |
---|---|---|---|
values | []string |
values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. |
false |
key | string |
key is the label key that the selector applies to. |
true |
operator | string |
operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. |
true |
Plotter.spec.assets[key]
AssetDetails is a list of assets used in the fybrikapplication. In addition to assets declared in fybrikapplication, AssetDetails list also contains assets that are allocated by the control-plane in order to serve fybrikapplication
Name | Type | Description | Required |
---|---|---|---|
advertisedAssetId | string |
AdvertisedAssetID links this asset to asset from fybrikapplication and is used by user facing services |
false |
assetDetails | object |
DataStore contains the details for accesing the data that are sent by catalog connectors Credentials for accesing the data are stored in Vault, in the location represented by Vault property. |
true |
Plotter.spec.assets[key].assetDetails
DataStore contains the details for accesing the data that are sent by catalog connectors Credentials for accesing the data are stored in Vault, in the location represented by Vault property.
Name | Type | Description | Required |
---|---|---|---|
format | string |
Format represents data format (e.g. parquet) as received from catalog connectors |
false |
connection | object |
Connection has the relevant details for accesing the data (url, table, ssl, etc.) |
true |
vault | map[string]object |
Holds details for retrieving credentials by the modules from Vault store. It is a map so that different credentials can be stored for the different DataFlow operations. |
true |
Plotter.spec.assets[key].assetDetails.connection
Connection has the relevant details for accesing the data (url, table, ssl, etc.)
Name | Type | Description | Required |
---|---|---|---|
name | string |
|
true |
Plotter.spec.assets[key].assetDetails.vault[key]
Holds details for retrieving credentials from Vault store.
Name | Type | Description | Required |
---|---|---|---|
address | string |
Address is Vault address |
true |
authPath | string |
AuthPath is the path to auth method i.e. kubernetes |
true |
role | string |
Role is the Vault role used for retrieving the credentials |
true |
secretPath | string |
SecretPath is the path of the secret holding the Credentials in Vault |
true |
Plotter.spec.flows[index]
Flows is the list of data flows driven from fybrikapplication: Each element in the list holds the flow of the data requested in fybrikapplication.
Name | Type | Description | Required |
---|---|---|---|
assetId | string |
AssetID indicates the data set being used in this data flow |
true |
flowType | string |
Type of the flow (e.g. read) |
true |
name | string |
Name of the flow |
true |
subFlows | []object |
|
true |
Plotter.spec.flows[index].subFlows[index]
Subflows is a list of data flows which are originated from the same data asset but are triggered differently (e.g., one upon init trigger and one upon workload trigger)
Name | Type | Description | Required |
---|---|---|---|
flowType | string |
Type of the flow (e.g. read) |
true |
name | string |
Name of the SubFlow |
true |
steps | [][]object |
Steps defines a series of sequential/parallel data flow steps The first dimension represents parallel data flows. The second sequential components within the same parallel data flow. |
true |
triggers | []enum |
Triggers |
true |
Plotter.spec.flows[index].subFlows[index].steps[index][index]
DataFlowStep contains details on a single data flow step
Name | Type | Description | Required |
---|---|---|---|
parameters | object |
Step parameters TODO why not flatten the parameters into this data flow step |
false |
cluster | string |
Name of the cluster this step is executed on |
true |
name | string |
Name of the step |
true |
template | string |
Template is the name of the template to execute the step The full details of the template can be extracted from Plotter.spec.templates list field. |
true |
Plotter.spec.flows[index].subFlows[index].steps[index][index].parameters
Step parameters TODO why not flatten the parameters into this data flow step
Name | Type | Description | Required |
---|---|---|---|
action | []object |
Actions are the data transformations that the module supports |
false |
api | object |
ResourceDetails includes asset connection details |
false |
sink | object |
StepSink holds information to where the target data will be written: it could be assetID of an asset specified in fybrikapplication or of an asset created by fybrik control-plane |
false |
source | object |
StepSource is the source of this step: it could be assetID or an enpoint of another step |
false |
Plotter.spec.flows[index].subFlows[index].steps[index][index].parameters.action[index]
Name | Type | Description | Required |
---|---|---|---|
name | string |
|
true |
Plotter.spec.flows[index].subFlows[index].steps[index][index].parameters.api
ResourceDetails includes asset connection details
Name | Type | Description | Required |
---|---|---|---|
dataFormat | string |
Data format |
false |
connection | object |
Connection information |
true |
Plotter.spec.flows[index].subFlows[index].steps[index][index].parameters.api.connection
Connection information
Name | Type | Description | Required |
---|---|---|---|
name | string |
|
true |
Plotter.spec.flows[index].subFlows[index].steps[index][index].parameters.sink
StepSink holds information to where the target data will be written: it could be assetID of an asset specified in fybrikapplication or of an asset created by fybrik control-plane
Name | Type | Description | Required |
---|---|---|---|
assetId | string |
AssetID identifies the target asset of this step |
true |
Plotter.spec.flows[index].subFlows[index].steps[index][index].parameters.source
StepSource is the source of this step: it could be assetID or an enpoint of another step
Name | Type | Description | Required |
---|---|---|---|
api | object |
API holds information for accessing a module instance |
false |
assetId | string |
AssetID identifies the source asset of this step |
false |
Plotter.spec.flows[index].subFlows[index].steps[index][index].parameters.source.api
API holds information for accessing a module instance
Name | Type | Description | Required |
---|---|---|---|
dataFormat | string |
Data format |
false |
connection | object |
Connection information |
true |
Plotter.spec.flows[index].subFlows[index].steps[index][index].parameters.source.api.connection
Connection information
Name | Type | Description | Required |
---|---|---|---|
name | string |
|
true |
Plotter.spec.templates[key]
Template contains basic information about the required modules to serve the fybrikapplication e.g., the module helm chart name.
Name | Type | Description | Required |
---|---|---|---|
name | string |
Name of the template |
false |
modules | []object |
Modules is a list of dependent modules. e.g., if a plugin module is used then the service module is used in should appear first in the modules list of the same template. If the modules list contains more than one module, the first module in the list is referred to as the "primary module" of which all the parameters to this template are sent to. |
true |
Plotter.spec.templates[key].modules[index]
ModuleInfo is a copy of FybrikModule Custom Resource. It contains information to instantiate resource of type FybrikModule.
Name | Type | Description | Required |
---|---|---|---|
scope | enum |
Scope indicates at what level the capability is used: workload, asset, cluster If not indicated it is assumed to be asset Enum: asset, workload, cluster |
false |
chart | object |
Chart contains the information needed to use helm to install the capability |
true |
name | string |
Name of the module |
true |
type | string |
May be one of service, config or plugin Service: Means that the control plane deploys the component that performs the capability Config: Another pre-installed service performs the capability and the module deployed configures it for the particular workload or dataset Plugin: Indicates that this module performs a capability as part of another service or module rather than as a stand-alone module |
true |
Plotter.spec.templates[key].modules[index].chart
Chart contains the information needed to use helm to install the capability
Name | Type | Description | Required |
---|---|---|---|
chartPullSecret | string |
Name of secret containing helm registry credentials |
false |
values | map[string]string |
Values to pass to helm chart installation |
false |
name | string |
Name of helm chart |
true |
Plotter.status
PlotterStatus defines the observed state of Plotter This includes readiness, error message, and indicators received from blueprint resources owned by the Plotter for cleanup and status monitoring
Name | Type | Description | Required |
---|---|---|---|
assets | map[string]object |
Assets is a map containing the status per asset. The key of this map is assetId |
false |
blueprints | map[string]object |
|
false |
conditions | []object |
Conditions represent the possible error and failure conditions |
false |
flows | map[string]object |
Flows is a map containing the status for each flow the key is the flow name |
false |
observedGeneration | integer |
ObservedGeneration is taken from the Plotter metadata. This is used to determine during reconcile whether reconcile was called because the desired state changed, or whether status of the allocated blueprints should be checked. Format: int64 |
false |
observedState | object |
ObservedState includes information to be reported back to the FybrikApplication resource It includes readiness and error indications, as well as user instructions |
false |
readyTimestamp | string |
Format: date-time |
false |
Plotter.status.assets[key]
ObservedState represents a part of the generated Blueprint/Plotter resource status that allows update of FybrikApplication status
Name | Type | Description | Required |
---|---|---|---|
error | string |
Error indicates that there has been an error to orchestrate the modules and provides the error message |
false |
ready | boolean |
Ready represents that the modules have been orchestrated successfully and the data is ready for usage |
false |
Plotter.status.blueprints[key]
MetaBlueprint defines blueprint metadata (name, namespace) and status
Name | Type | Description | Required |
---|---|---|---|
name | string |
|
true |
namespace | string |
|
true |
status | object |
BlueprintStatus defines the observed state of Blueprint This includes readiness, error message, and indicators for the Kubernetes resources owned by the Blueprint for cleanup and status monitoring |
true |
Plotter.status.blueprints[key].status
BlueprintStatus defines the observed state of Blueprint This includes readiness, error message, and indicators for the Kubernetes resources owned by the Blueprint for cleanup and status monitoring
Name | Type | Description | Required |
---|---|---|---|
modules | map[string]object |
ModulesState is a map which holds the status of each module its key is the moduleInstanceName which is the unique name for the deployed instance related to this workload |
false |
observedGeneration | integer |
ObservedGeneration is taken from the Blueprint metadata. This is used to determine during reconcile whether reconcile was called because the desired state changed, or whether status of the allocated resources should be checked. Format: int64 |
false |
observedState | object |
ObservedState includes information to be reported back to the FybrikApplication resource It includes readiness and error indications, as well as user instructions |
false |
releases | map[string]integer |
Releases map each release to the observed generation of the blueprint containing this release. At the end of reconcile, each release should be mapped to the latest blueprint version or be uninstalled. |
false |
Plotter.status.blueprints[key].status.modules[key]
ObservedState represents a part of the generated Blueprint/Plotter resource status that allows update of FybrikApplication status
Name | Type | Description | Required |
---|---|---|---|
error | string |
Error indicates that there has been an error to orchestrate the modules and provides the error message |
false |
ready | boolean |
Ready represents that the modules have been orchestrated successfully and the data is ready for usage |
false |
Plotter.status.blueprints[key].status.observedState
ObservedState includes information to be reported back to the FybrikApplication resource It includes readiness and error indications, as well as user instructions
Name | Type | Description | Required |
---|---|---|---|
error | string |
Error indicates that there has been an error to orchestrate the modules and provides the error message |
false |
ready | boolean |
Ready represents that the modules have been orchestrated successfully and the data is ready for usage |
false |
Plotter.status.conditions[index]
Condition describes the state of a FybrikApplication at a certain point.
Name | Type | Description | Required |
---|---|---|---|
message | string |
Message contains the details of the current condition |
false |
observedGeneration | integer |
ObservedGeneration is the version of the resource for which the condition has been evaluated Format: int64 |
false |
status | enum |
Status of the condition, one of (`True`, `False`, `Unknown`). Enum: True, False, Unknown Default: Unknown |
false |
type | string |
Type of the condition |
true |
Plotter.status.flows[key]
FlowStatus includes information to be reported back to the FybrikApplication resource It holds the status per data flow
Name | Type | Description | Required |
---|---|---|---|
status | object |
ObservedState includes information about the current flow It includes readiness and error indications, as well as user instructions |
false |
subFlows | map[string]object |
|
true |
Plotter.status.flows[key].status
ObservedState includes information about the current flow It includes readiness and error indications, as well as user instructions
Name | Type | Description | Required |
---|---|---|---|
error | string |
Error indicates that there has been an error to orchestrate the modules and provides the error message |
false |
ready | boolean |
Ready represents that the modules have been orchestrated successfully and the data is ready for usage |
false |
Plotter.status.flows[key].subFlows[key]
ObservedState represents a part of the generated Blueprint/Plotter resource status that allows update of FybrikApplication status
Name | Type | Description | Required |
---|---|---|---|
error | string |
Error indicates that there has been an error to orchestrate the modules and provides the error message |
false |
ready | boolean |
Ready represents that the modules have been orchestrated successfully and the data is ready for usage |
false |
Plotter.status.observedState
ObservedState includes information to be reported back to the FybrikApplication resource It includes readiness and error indications, as well as user instructions
Name | Type | Description | Required |
---|---|---|---|
error | string |
Error indicates that there has been an error to orchestrate the modules and provides the error message |
false |
ready | boolean |
Ready represents that the modules have been orchestrated successfully and the data is ready for usage |
false |
katalog.fybrik.io/v1alpha1
Resource Types:
Asset
Asset defines an asset in the catalog
Name | Type | Description | Required |
---|---|---|---|
apiVersion | string | katalog.fybrik.io/v1alpha1 | true |
kind | string | Asset | true |
metadata | object | Refer to the Kubernetes API documentation for the fields of the `metadata` field. | true |
spec | object |
|
false |
Asset.spec
Name | Type | Description | Required |
---|---|---|---|
details | object |
Asset details |
true |
metadata | object |
Asset metadata |
true |
secretRef | object |
Reference to a Secret resource holding credentials for this asset |
true |
Asset.spec.details
Asset details
Name | Type | Description | Required |
---|---|---|---|
dataFormat | string |
Data format |
false |
connection | object |
Connection information |
true |
Asset.spec.details.connection
Connection information
Name | Type | Description | Required |
---|---|---|---|
name | string |
|
true |
Asset.spec.metadata
Asset metadata
Name | Type | Description | Required |
---|---|---|---|
columns | []object |
Columns associated with the asset |
false |
geography | string |
Geography of the resource |
false |
name | string |
Name of the resource |
false |
owner | string |
Owner of the resource |
false |
tags | object |
Tags associated with the asset |
false |
Asset.spec.metadata.columns[index]
ResourceColumn represents a column in a tabular resource
Name | Type | Description | Required |
---|---|---|---|
tags | object |
Tags associated with the column |
false |
name | string |
Name of the column |
true |
Asset.spec.secretRef
Reference to a Secret resource holding credentials for this asset
Name | Type | Description | Required |
---|---|---|---|
name | string |
Name of the Secret resource (must exist in the same namespace) |
true |