Sign in Dashboard Get started
  1. Packages
  2. Packages
  3. Databricks Provider
  4. API Docs
  5. FeatureEngineeringKafkaConfig
Viewing docs for Databricks v1.96.0
published on Thursday, Jun 18, 2026 by Pulumi
Schema (JSON)
pulumi/pulumi-databricks

databricks.FeatureEngineeringKafkaConfig

databricks logo
Viewing docs for Databricks v1.96.0
published on Thursday, Jun 18, 2026 by Pulumi
Schema (JSON)
pulumi/pulumi-databricks

    Private Preview

    Create FeatureEngineeringKafkaConfig Resource

    Resources are created with functions called constructors. To learn more about declaring and configuring resources, see Resources.

    Constructor syntax

    new FeatureEngineeringKafkaConfig(name: string, args: FeatureEngineeringKafkaConfigArgs, opts?: CustomResourceOptions);
    @overload
def FeatureEngineeringKafkaConfig(resource_name: str,
                                  args: FeatureEngineeringKafkaConfigArgs,
                                  opts: Optional[ResourceOptions] = None)

@overload
def FeatureEngineeringKafkaConfig(resource_name: str,
                                  opts: Optional[ResourceOptions] = None,
                                  auth_config: Optional[FeatureEngineeringKafkaConfigAuthConfigArgs] = None,
                                  bootstrap_servers: Optional[str] = None,
                                  subscription_mode: Optional[FeatureEngineeringKafkaConfigSubscriptionModeArgs] = None,
                                  backfill_source: Optional[FeatureEngineeringKafkaConfigBackfillSourceArgs] = None,
                                  extra_options: Optional[Mapping[str, str]] = None,
                                  ingestion_config: Optional[FeatureEngineeringKafkaConfigIngestionConfigArgs] = None,
                                  key_schema: Optional[FeatureEngineeringKafkaConfigKeySchemaArgs] = None,
                                  provider_config: Optional[FeatureEngineeringKafkaConfigProviderConfigArgs] = None,
                                  value_schema: Optional[FeatureEngineeringKafkaConfigValueSchemaArgs] = None)
    func NewFeatureEngineeringKafkaConfig(ctx *Context, name string, args FeatureEngineeringKafkaConfigArgs, opts ...ResourceOption) (*FeatureEngineeringKafkaConfig, error)
    public FeatureEngineeringKafkaConfig(string name, FeatureEngineeringKafkaConfigArgs args, CustomResourceOptions? opts = null)
    public FeatureEngineeringKafkaConfig(String name, FeatureEngineeringKafkaConfigArgs args)
public FeatureEngineeringKafkaConfig(String name, FeatureEngineeringKafkaConfigArgs args, CustomResourceOptions options)

    type: databricks:FeatureEngineeringKafkaConfig
properties: # The arguments to resource properties.
options: # Bag of options to control resource's behavior.


    resource "databricks_featureengineeringkafkaconfig" "name" {
    # resource properties
}

    Parameters

    name string
    The unique name of the resource.
    args FeatureEngineeringKafkaConfigArgs
    The arguments to resource properties.
    opts CustomResourceOptions
    Bag of options to control resource's behavior.
    resource_name str
    The unique name of the resource.
    args FeatureEngineeringKafkaConfigArgs
    The arguments to resource properties.
    opts ResourceOptions
    Bag of options to control resource's behavior.
    ctx Context
    Context object for the current deployment.
    name string
    The unique name of the resource.
    args FeatureEngineeringKafkaConfigArgs
    The arguments to resource properties.
    opts ResourceOption
    Bag of options to control resource's behavior.
    name string
    The unique name of the resource.
    args FeatureEngineeringKafkaConfigArgs
    The arguments to resource properties.
    opts CustomResourceOptions
    Bag of options to control resource's behavior.
    name String
    The unique name of the resource.
    args FeatureEngineeringKafkaConfigArgs
    The arguments to resource properties.
    options CustomResourceOptions
    Bag of options to control resource's behavior.

    Constructor example

    The following reference example uses placeholder values for all input properties.

    var featureEngineeringKafkaConfigResource = new Databricks.FeatureEngineeringKafkaConfig("featureEngineeringKafkaConfigResource", new()
{
    AuthConfig = new Databricks.Inputs.FeatureEngineeringKafkaConfigAuthConfigArgs
    {
        MtlsConfig = new Databricks.Inputs.FeatureEngineeringKafkaConfigAuthConfigMtlsConfigArgs
        {
            KeyPasswordRef = new Databricks.Inputs.FeatureEngineeringKafkaConfigAuthConfigMtlsConfigKeyPasswordRefArgs
            {
                Key = "string",
                Scope = "string",
            },
            KeystoreLocation = "string",
            KeystorePasswordRef = new Databricks.Inputs.FeatureEngineeringKafkaConfigAuthConfigMtlsConfigKeystorePasswordRefArgs
            {
                Key = "string",
                Scope = "string",
            },
            TruststoreLocation = "string",
            TruststorePasswordRef = new Databricks.Inputs.FeatureEngineeringKafkaConfigAuthConfigMtlsConfigTruststorePasswordRefArgs
            {
                Key = "string",
                Scope = "string",
            },
            DisableHostnameVerification = false,
        },
        UcServiceCredentialName = "string",
    },
    BootstrapServers = "string",
    SubscriptionMode = new Databricks.Inputs.FeatureEngineeringKafkaConfigSubscriptionModeArgs
    {
        Assign = "string",
        Subscribe = "string",
        SubscribePattern = "string",
    },
    BackfillSource = new Databricks.Inputs.FeatureEngineeringKafkaConfigBackfillSourceArgs
    {
        DeltaTableName = "string",
        DeltaTableSource = new Databricks.Inputs.FeatureEngineeringKafkaConfigBackfillSourceDeltaTableSourceArgs
        {
            FullName = "string",
            DataframeSchema = "string",
            EntityColumns = new[]
            {
                "string",
            },
            FilterCondition = "string",
            TimeseriesColumn = "string",
            TransformationSql = "string",
        },
    },
    ExtraOptions = 
    {
        { "string", "string" },
    },
    IngestionConfig = new Databricks.Inputs.FeatureEngineeringKafkaConfigIngestionConfigArgs
    {
        IngestionDestination = new Databricks.Inputs.FeatureEngineeringKafkaConfigIngestionConfigIngestionDestinationArgs
        {
            DeltaTableName = "string",
        },
        BackfillJobId = 0,
        BackfillSource = new Databricks.Inputs.FeatureEngineeringKafkaConfigIngestionConfigBackfillSourceArgs
        {
            DeltaTableName = "string",
            DeltaTableSource = new Databricks.Inputs.FeatureEngineeringKafkaConfigIngestionConfigBackfillSourceDeltaTableSourceArgs
            {
                FullName = "string",
                DataframeSchema = "string",
                EntityColumns = new[]
                {
                    "string",
                },
                FilterCondition = "string",
                TimeseriesColumn = "string",
                TransformationSql = "string",
            },
        },
        DeduplicationColumns = new[]
        {
            "string",
        },
        IngestionJobId = 0,
        IngestionPipelineId = "string",
    },
    KeySchema = new Databricks.Inputs.FeatureEngineeringKafkaConfigKeySchemaArgs
    {
        JsonSchema = "string",
    },
    ProviderConfig = new Databricks.Inputs.FeatureEngineeringKafkaConfigProviderConfigArgs
    {
        WorkspaceId = "string",
    },
    ValueSchema = new Databricks.Inputs.FeatureEngineeringKafkaConfigValueSchemaArgs
    {
        JsonSchema = "string",
    },
});

    example, err := databricks.NewFeatureEngineeringKafkaConfig(ctx, "featureEngineeringKafkaConfigResource", &databricks.FeatureEngineeringKafkaConfigArgs{
	AuthConfig: &databricks.FeatureEngineeringKafkaConfigAuthConfigArgs{
		MtlsConfig: &databricks.FeatureEngineeringKafkaConfigAuthConfigMtlsConfigArgs{
			KeyPasswordRef: &databricks.FeatureEngineeringKafkaConfigAuthConfigMtlsConfigKeyPasswordRefArgs{
				Key:   pulumi.String("string"),
				Scope: pulumi.String("string"),
			},
			KeystoreLocation: pulumi.String("string"),
			KeystorePasswordRef: &databricks.FeatureEngineeringKafkaConfigAuthConfigMtlsConfigKeystorePasswordRefArgs{
				Key:   pulumi.String("string"),
				Scope: pulumi.String("string"),
			},
			TruststoreLocation: pulumi.String("string"),
			TruststorePasswordRef: &databricks.FeatureEngineeringKafkaConfigAuthConfigMtlsConfigTruststorePasswordRefArgs{
				Key:   pulumi.String("string"),
				Scope: pulumi.String("string"),
			},
			DisableHostnameVerification: pulumi.Bool(false),
		},
		UcServiceCredentialName: pulumi.String("string"),
	},
	BootstrapServers: pulumi.String("string"),
	SubscriptionMode: &databricks.FeatureEngineeringKafkaConfigSubscriptionModeArgs{
		Assign:           pulumi.String("string"),
		Subscribe:        pulumi.String("string"),
		SubscribePattern: pulumi.String("string"),
	},
	BackfillSource: &databricks.FeatureEngineeringKafkaConfigBackfillSourceArgs{
		DeltaTableName: pulumi.String("string"),
		DeltaTableSource: &databricks.FeatureEngineeringKafkaConfigBackfillSourceDeltaTableSourceArgs{
			FullName:        pulumi.String("string"),
			DataframeSchema: pulumi.String("string"),
			EntityColumns: pulumi.StringArray{
				pulumi.String("string"),
			},
			FilterCondition:   pulumi.String("string"),
			TimeseriesColumn:  pulumi.String("string"),
			TransformationSql: pulumi.String("string"),
		},
	},
	ExtraOptions: pulumi.StringMap{
		"string": pulumi.String("string"),
	},
	IngestionConfig: &databricks.FeatureEngineeringKafkaConfigIngestionConfigArgs{
		IngestionDestination: &databricks.FeatureEngineeringKafkaConfigIngestionConfigIngestionDestinationArgs{
			DeltaTableName: pulumi.String("string"),
		},
		BackfillJobId: pulumi.Int(0),
		BackfillSource: &databricks.FeatureEngineeringKafkaConfigIngestionConfigBackfillSourceArgs{
			DeltaTableName: pulumi.String("string"),
			DeltaTableSource: &databricks.FeatureEngineeringKafkaConfigIngestionConfigBackfillSourceDeltaTableSourceArgs{
				FullName:        pulumi.String("string"),
				DataframeSchema: pulumi.String("string"),
				EntityColumns: pulumi.StringArray{
					pulumi.String("string"),
				},
				FilterCondition:   pulumi.String("string"),
				TimeseriesColumn:  pulumi.String("string"),
				TransformationSql: pulumi.String("string"),
			},
		},
		DeduplicationColumns: pulumi.StringArray{
			pulumi.String("string"),
		},
		IngestionJobId:      pulumi.Int(0),
		IngestionPipelineId: pulumi.String("string"),
	},
	KeySchema: &databricks.FeatureEngineeringKafkaConfigKeySchemaArgs{
		JsonSchema: pulumi.String("string"),
	},
	ProviderConfig: &databricks.FeatureEngineeringKafkaConfigProviderConfigArgs{
		WorkspaceId: pulumi.String("string"),
	},
	ValueSchema: &databricks.FeatureEngineeringKafkaConfigValueSchemaArgs{
		JsonSchema: pulumi.String("string"),
	},
})

    resource "databricks_featureengineeringkafkaconfig" "featureEngineeringKafkaConfigResource" {
  auth_config = {
    mtls_config = {
      key_password_ref = {
        key   = "string"
        scope = "string"
      }
      keystore_location = "string"
      keystore_password_ref = {
        key   = "string"
        scope = "string"
      }
      truststore_location = "string"
      truststore_password_ref = {
        key   = "string"
        scope = "string"
      }
      disable_hostname_verification = false
    }
    uc_service_credential_name = "string"
  }
  bootstrap_servers = "string"
  subscription_mode = {
    assign            = "string"
    subscribe         = "string"
    subscribe_pattern = "string"
  }
  backfill_source = {
    delta_table_name = "string"
    delta_table_source = {
      full_name          = "string"
      dataframe_schema   = "string"
      entity_columns     = ["string"]
      filter_condition   = "string"
      timeseries_column  = "string"
      transformation_sql = "string"
    }
  }
  extra_options = {
    "string" = "string"
  }
  ingestion_config = {
    ingestion_destination = {
      delta_table_name = "string"
    }
    backfill_job_id = 0
    backfill_source = {
      delta_table_name = "string"
      delta_table_source = {
        full_name          = "string"
        dataframe_schema   = "string"
        entity_columns     = ["string"]
        filter_condition   = "string"
        timeseries_column  = "string"
        transformation_sql = "string"
      }
    }
    deduplication_columns = ["string"]
    ingestion_job_id      = 0
    ingestion_pipeline_id = "string"
  }
  key_schema = {
    json_schema = "string"
  }
  provider_config = {
    workspace_id = "string"
  }
  value_schema = {
    json_schema = "string"
  }
}

    var featureEngineeringKafkaConfigResource = new FeatureEngineeringKafkaConfig("featureEngineeringKafkaConfigResource", FeatureEngineeringKafkaConfigArgs.builder()
    .authConfig(FeatureEngineeringKafkaConfigAuthConfigArgs.builder()
        .mtlsConfig(FeatureEngineeringKafkaConfigAuthConfigMtlsConfigArgs.builder()
            .keyPasswordRef(FeatureEngineeringKafkaConfigAuthConfigMtlsConfigKeyPasswordRefArgs.builder()
                .key("string")
                .scope("string")
                .build())
            .keystoreLocation("string")
            .keystorePasswordRef(FeatureEngineeringKafkaConfigAuthConfigMtlsConfigKeystorePasswordRefArgs.builder()
                .key("string")
                .scope("string")
                .build())
            .truststoreLocation("string")
            .truststorePasswordRef(FeatureEngineeringKafkaConfigAuthConfigMtlsConfigTruststorePasswordRefArgs.builder()
                .key("string")
                .scope("string")
                .build())
            .disableHostnameVerification(false)
            .build())
        .ucServiceCredentialName("string")
        .build())
    .bootstrapServers("string")
    .subscriptionMode(FeatureEngineeringKafkaConfigSubscriptionModeArgs.builder()
        .assign("string")
        .subscribe("string")
        .subscribePattern("string")
        .build())
    .backfillSource(FeatureEngineeringKafkaConfigBackfillSourceArgs.builder()
        .deltaTableName("string")
        .deltaTableSource(FeatureEngineeringKafkaConfigBackfillSourceDeltaTableSourceArgs.builder()
            .fullName("string")
            .dataframeSchema("string")
            .entityColumns("string")
            .filterCondition("string")
            .timeseriesColumn("string")
            .transformationSql("string")
            .build())
        .build())
    .extraOptions(Map.of("string", "string"))
    .ingestionConfig(FeatureEngineeringKafkaConfigIngestionConfigArgs.builder()
        .ingestionDestination(FeatureEngineeringKafkaConfigIngestionConfigIngestionDestinationArgs.builder()
            .deltaTableName("string")
            .build())
        .backfillJobId(0)
        .backfillSource(FeatureEngineeringKafkaConfigIngestionConfigBackfillSourceArgs.builder()
            .deltaTableName("string")
            .deltaTableSource(FeatureEngineeringKafkaConfigIngestionConfigBackfillSourceDeltaTableSourceArgs.builder()
                .fullName("string")
                .dataframeSchema("string")
                .entityColumns("string")
                .filterCondition("string")
                .timeseriesColumn("string")
                .transformationSql("string")
                .build())
            .build())
        .deduplicationColumns("string")
        .ingestionJobId(0)
        .ingestionPipelineId("string")
        .build())
    .keySchema(FeatureEngineeringKafkaConfigKeySchemaArgs.builder()
        .jsonSchema("string")
        .build())
    .providerConfig(FeatureEngineeringKafkaConfigProviderConfigArgs.builder()
        .workspaceId("string")
        .build())
    .valueSchema(FeatureEngineeringKafkaConfigValueSchemaArgs.builder()
        .jsonSchema("string")
        .build())
    .build());

    feature_engineering_kafka_config_resource = databricks.FeatureEngineeringKafkaConfig("featureEngineeringKafkaConfigResource",
    auth_config={
        "mtls_config": {
            "key_password_ref": {
                "key": "string",
                "scope": "string",
            },
            "keystore_location": "string",
            "keystore_password_ref": {
                "key": "string",
                "scope": "string",
            },
            "truststore_location": "string",
            "truststore_password_ref": {
                "key": "string",
                "scope": "string",
            },
            "disable_hostname_verification": False,
        },
        "uc_service_credential_name": "string",
    },
    bootstrap_servers="string",
    subscription_mode={
        "assign": "string",
        "subscribe": "string",
        "subscribe_pattern": "string",
    },
    backfill_source={
        "delta_table_name": "string",
        "delta_table_source": {
            "full_name": "string",
            "dataframe_schema": "string",
            "entity_columns": ["string"],
            "filter_condition": "string",
            "timeseries_column": "string",
            "transformation_sql": "string",
        },
    },
    extra_options={
        "string": "string",
    },
    ingestion_config={
        "ingestion_destination": {
            "delta_table_name": "string",
        },
        "backfill_job_id": 0,
        "backfill_source": {
            "delta_table_name": "string",
            "delta_table_source": {
                "full_name": "string",
                "dataframe_schema": "string",
                "entity_columns": ["string"],
                "filter_condition": "string",
                "timeseries_column": "string",
                "transformation_sql": "string",
            },
        },
        "deduplication_columns": ["string"],
        "ingestion_job_id": 0,
        "ingestion_pipeline_id": "string",
    },
    key_schema={
        "json_schema": "string",
    },
    provider_config={
        "workspace_id": "string",
    },
    value_schema={
        "json_schema": "string",
    })

    const featureEngineeringKafkaConfigResource = new databricks.FeatureEngineeringKafkaConfig("featureEngineeringKafkaConfigResource", {
    authConfig: {
        mtlsConfig: {
            keyPasswordRef: {
                key: "string",
                scope: "string",
            },
            keystoreLocation: "string",
            keystorePasswordRef: {
                key: "string",
                scope: "string",
            },
            truststoreLocation: "string",
            truststorePasswordRef: {
                key: "string",
                scope: "string",
            },
            disableHostnameVerification: false,
        },
        ucServiceCredentialName: "string",
    },
    bootstrapServers: "string",
    subscriptionMode: {
        assign: "string",
        subscribe: "string",
        subscribePattern: "string",
    },
    backfillSource: {
        deltaTableName: "string",
        deltaTableSource: {
            fullName: "string",
            dataframeSchema: "string",
            entityColumns: ["string"],
            filterCondition: "string",
            timeseriesColumn: "string",
            transformationSql: "string",
        },
    },
    extraOptions: {
        string: "string",
    },
    ingestionConfig: {
        ingestionDestination: {
            deltaTableName: "string",
        },
        backfillJobId: 0,
        backfillSource: {
            deltaTableName: "string",
            deltaTableSource: {
                fullName: "string",
                dataframeSchema: "string",
                entityColumns: ["string"],
                filterCondition: "string",
                timeseriesColumn: "string",
                transformationSql: "string",
            },
        },
        deduplicationColumns: ["string"],
        ingestionJobId: 0,
        ingestionPipelineId: "string",
    },
    keySchema: {
        jsonSchema: "string",
    },
    providerConfig: {
        workspaceId: "string",
    },
    valueSchema: {
        jsonSchema: "string",
    },
});

    type: databricks:FeatureEngineeringKafkaConfig
properties:
    authConfig:
        mtlsConfig:
            disableHostnameVerification: false
            keyPasswordRef:
                key: string
                scope: string
            keystoreLocation: string
            keystorePasswordRef:
                key: string
                scope: string
            truststoreLocation: string
            truststorePasswordRef:
                key: string
                scope: string
        ucServiceCredentialName: string
    backfillSource:
        deltaTableName: string
        deltaTableSource:
            dataframeSchema: string
            entityColumns:
                - string
            filterCondition: string
            fullName: string
            timeseriesColumn: string
            transformationSql: string
    bootstrapServers: string
    extraOptions:
        string: string
    ingestionConfig:
        backfillJobId: 0
        backfillSource:
            deltaTableName: string
            deltaTableSource:
                dataframeSchema: string
                entityColumns:
                    - string
                filterCondition: string
                fullName: string
                timeseriesColumn: string
                transformationSql: string
        deduplicationColumns:
            - string
        ingestionDestination:
            deltaTableName: string
        ingestionJobId: 0
        ingestionPipelineId: string
    keySchema:
        jsonSchema: string
    providerConfig:
        workspaceId: string
    subscriptionMode:
        assign: string
        subscribe: string
        subscribePattern: string
    valueSchema:
        jsonSchema: string

    FeatureEngineeringKafkaConfig Resource Properties

    To learn more about resource properties and how to use them, see Inputs and Outputs in the Architecture and Concepts docs.

    Inputs

    In Python, inputs that are objects can be passed either as argument classes or as dictionary literals.

    The FeatureEngineeringKafkaConfig resource accepts the following input properties:

    AuthConfig FeatureEngineeringKafkaConfigAuthConfig
    Authentication configuration for connection to topics
    BootstrapServers string
    A comma-separated list of host/port pairs pointing to Kafka cluster
    SubscriptionMode FeatureEngineeringKafkaConfigSubscriptionMode
    Options to configure which Kafka topics to pull data from
    BackfillSource FeatureEngineeringKafkaConfigBackfillSource
    A user-provided and managed source for backfilling data. Historical data is used when creating a training set from streaming features linked to this Kafka config. In the future, a separate table will be maintained by Databricks for forward filling data. The schema for this source must match exactly that of the key and value schemas specified for this Kafka config
    ExtraOptions Dictionary<string, string>
    Catch-all for miscellaneous options. Keys should be source options or Kafka consumer options (kafka.*)
    IngestionConfig FeatureEngineeringKafkaConfigIngestionConfig
    Configuration for ingesting Kafka data into a Databricks-managed Delta table
    KeySchema FeatureEngineeringKafkaConfigKeySchema
    Schema configuration for extracting message keys from topics. At least one of keySchema and valueSchema must be provided
    ProviderConfig FeatureEngineeringKafkaConfigProviderConfig
    Configure the provider for management through account provider.
    ValueSchema FeatureEngineeringKafkaConfigValueSchema
    Schema configuration for extracting message values from topics. At least one of keySchema and valueSchema must be provided
    AuthConfig FeatureEngineeringKafkaConfigAuthConfigArgs
    Authentication configuration for connection to topics
    BootstrapServers string
    A comma-separated list of host/port pairs pointing to Kafka cluster
    SubscriptionMode FeatureEngineeringKafkaConfigSubscriptionModeArgs
    Options to configure which Kafka topics to pull data from
    BackfillSource FeatureEngineeringKafkaConfigBackfillSourceArgs
    A user-provided and managed source for backfilling data. Historical data is used when creating a training set from streaming features linked to this Kafka config. In the future, a separate table will be maintained by Databricks for forward filling data. The schema for this source must match exactly that of the key and value schemas specified for this Kafka config
    ExtraOptions map[string]string
    Catch-all for miscellaneous options. Keys should be source options or Kafka consumer options (kafka.*)
    IngestionConfig FeatureEngineeringKafkaConfigIngestionConfigArgs
    Configuration for ingesting Kafka data into a Databricks-managed Delta table
    KeySchema FeatureEngineeringKafkaConfigKeySchemaArgs
    Schema configuration for extracting message keys from topics. At least one of keySchema and valueSchema must be provided
    ProviderConfig FeatureEngineeringKafkaConfigProviderConfigArgs
    Configure the provider for management through account provider.
    ValueSchema FeatureEngineeringKafkaConfigValueSchemaArgs
    Schema configuration for extracting message values from topics. At least one of keySchema and valueSchema must be provided
    auth_config object
    Authentication configuration for connection to topics
    bootstrap_servers string
    A comma-separated list of host/port pairs pointing to Kafka cluster
    subscription_mode object
    Options to configure which Kafka topics to pull data from
    backfill_source object
    A user-provided and managed source for backfilling data. Historical data is used when creating a training set from streaming features linked to this Kafka config. In the future, a separate table will be maintained by Databricks for forward filling data. The schema for this source must match exactly that of the key and value schemas specified for this Kafka config
    extra_options map(string)
    Catch-all for miscellaneous options. Keys should be source options or Kafka consumer options (kafka.*)
    ingestion_config object
    Configuration for ingesting Kafka data into a Databricks-managed Delta table
    key_schema object
    Schema configuration for extracting message keys from topics. At least one of keySchema and valueSchema must be provided
    provider_config object
    Configure the provider for management through account provider.
    value_schema object
    Schema configuration for extracting message values from topics. At least one of keySchema and valueSchema must be provided
    authConfig FeatureEngineeringKafkaConfigAuthConfig
    Authentication configuration for connection to topics
    bootstrapServers String
    A comma-separated list of host/port pairs pointing to Kafka cluster
    subscriptionMode FeatureEngineeringKafkaConfigSubscriptionMode
    Options to configure which Kafka topics to pull data from
    backfillSource FeatureEngineeringKafkaConfigBackfillSource
    A user-provided and managed source for backfilling data. Historical data is used when creating a training set from streaming features linked to this Kafka config. In the future, a separate table will be maintained by Databricks for forward filling data. The schema for this source must match exactly that of the key and value schemas specified for this Kafka config
    extraOptions Map<String,String>
    Catch-all for miscellaneous options. Keys should be source options or Kafka consumer options (kafka.*)
    ingestionConfig FeatureEngineeringKafkaConfigIngestionConfig
    Configuration for ingesting Kafka data into a Databricks-managed Delta table
    keySchema FeatureEngineeringKafkaConfigKeySchema
    Schema configuration for extracting message keys from topics. At least one of keySchema and valueSchema must be provided
    providerConfig FeatureEngineeringKafkaConfigProviderConfig
    Configure the provider for management through account provider.
    valueSchema FeatureEngineeringKafkaConfigValueSchema
    Schema configuration for extracting message values from topics. At least one of keySchema and valueSchema must be provided
    authConfig FeatureEngineeringKafkaConfigAuthConfig
    Authentication configuration for connection to topics
    bootstrapServers string
    A comma-separated list of host/port pairs pointing to Kafka cluster
    subscriptionMode FeatureEngineeringKafkaConfigSubscriptionMode
    Options to configure which Kafka topics to pull data from
    backfillSource FeatureEngineeringKafkaConfigBackfillSource
    A user-provided and managed source for backfilling data. Historical data is used when creating a training set from streaming features linked to this Kafka config. In the future, a separate table will be maintained by Databricks for forward filling data. The schema for this source must match exactly that of the key and value schemas specified for this Kafka config
    extraOptions {[key: string]: string}
    Catch-all for miscellaneous options. Keys should be source options or Kafka consumer options (kafka.*)
    ingestionConfig FeatureEngineeringKafkaConfigIngestionConfig
    Configuration for ingesting Kafka data into a Databricks-managed Delta table
    keySchema FeatureEngineeringKafkaConfigKeySchema
    Schema configuration for extracting message keys from topics. At least one of keySchema and valueSchema must be provided
    providerConfig FeatureEngineeringKafkaConfigProviderConfig
    Configure the provider for management through account provider.
    valueSchema FeatureEngineeringKafkaConfigValueSchema
    Schema configuration for extracting message values from topics. At least one of keySchema and valueSchema must be provided
    auth_config FeatureEngineeringKafkaConfigAuthConfigArgs
    Authentication configuration for connection to topics
    bootstrap_servers str
    A comma-separated list of host/port pairs pointing to Kafka cluster
    subscription_mode FeatureEngineeringKafkaConfigSubscriptionModeArgs
    Options to configure which Kafka topics to pull data from
    backfill_source FeatureEngineeringKafkaConfigBackfillSourceArgs
    A user-provided and managed source for backfilling data. Historical data is used when creating a training set from streaming features linked to this Kafka config. In the future, a separate table will be maintained by Databricks for forward filling data. The schema for this source must match exactly that of the key and value schemas specified for this Kafka config
    extra_options Mapping[str, str]
    Catch-all for miscellaneous options. Keys should be source options or Kafka consumer options (kafka.*)
    ingestion_config FeatureEngineeringKafkaConfigIngestionConfigArgs
    Configuration for ingesting Kafka data into a Databricks-managed Delta table
    key_schema FeatureEngineeringKafkaConfigKeySchemaArgs
    Schema configuration for extracting message keys from topics. At least one of keySchema and valueSchema must be provided
    provider_config FeatureEngineeringKafkaConfigProviderConfigArgs
    Configure the provider for management through account provider.
    value_schema FeatureEngineeringKafkaConfigValueSchemaArgs
    Schema configuration for extracting message values from topics. At least one of keySchema and valueSchema must be provided
    authConfig Property Map
    Authentication configuration for connection to topics
    bootstrapServers String
    A comma-separated list of host/port pairs pointing to Kafka cluster
    subscriptionMode Property Map
    Options to configure which Kafka topics to pull data from
    backfillSource Property Map
    A user-provided and managed source for backfilling data. Historical data is used when creating a training set from streaming features linked to this Kafka config. In the future, a separate table will be maintained by Databricks for forward filling data. The schema for this source must match exactly that of the key and value schemas specified for this Kafka config
    extraOptions Map<String>
    Catch-all for miscellaneous options. Keys should be source options or Kafka consumer options (kafka.*)
    ingestionConfig Property Map
    Configuration for ingesting Kafka data into a Databricks-managed Delta table
    keySchema Property Map
    Schema configuration for extracting message keys from topics. At least one of keySchema and valueSchema must be provided
    providerConfig Property Map
    Configure the provider for management through account provider.
    valueSchema Property Map
    Schema configuration for extracting message values from topics. At least one of keySchema and valueSchema must be provided

    Outputs

    All input properties are implicitly available as output properties. Additionally, the FeatureEngineeringKafkaConfig resource produces the following output properties:

    Id string
    The provider-assigned unique ID for this managed resource.
    Name string
    (string) - Name that uniquely identifies this Kafka config within the metastore. This will be the identifier used from the Feature object to reference these configs for a feature. Can be distinct from topic name
    Id string
    The provider-assigned unique ID for this managed resource.
    Name string
    (string) - Name that uniquely identifies this Kafka config within the metastore. This will be the identifier used from the Feature object to reference these configs for a feature. Can be distinct from topic name
    id string
    The provider-assigned unique ID for this managed resource.
    name string
    (string) - Name that uniquely identifies this Kafka config within the metastore. This will be the identifier used from the Feature object to reference these configs for a feature. Can be distinct from topic name
    id String
    The provider-assigned unique ID for this managed resource.
    name String
    (string) - Name that uniquely identifies this Kafka config within the metastore. This will be the identifier used from the Feature object to reference these configs for a feature. Can be distinct from topic name
    id string
    The provider-assigned unique ID for this managed resource.
    name string
    (string) - Name that uniquely identifies this Kafka config within the metastore. This will be the identifier used from the Feature object to reference these configs for a feature. Can be distinct from topic name
    id str
    The provider-assigned unique ID for this managed resource.
    name str
    (string) - Name that uniquely identifies this Kafka config within the metastore. This will be the identifier used from the Feature object to reference these configs for a feature. Can be distinct from topic name
    id String
    The provider-assigned unique ID for this managed resource.
    name String
    (string) - Name that uniquely identifies this Kafka config within the metastore. This will be the identifier used from the Feature object to reference these configs for a feature. Can be distinct from topic name

    Look up Existing FeatureEngineeringKafkaConfig Resource

    Get an existing FeatureEngineeringKafkaConfig resource’s state with the given name, ID, and optional extra properties used to qualify the lookup.

    public static get(name: string, id: Input<ID>, state?: FeatureEngineeringKafkaConfigState, opts?: CustomResourceOptions): FeatureEngineeringKafkaConfig
    @staticmethod
def get(resource_name: str,
        id: str,
        opts: Optional[ResourceOptions] = None,
        auth_config: Optional[FeatureEngineeringKafkaConfigAuthConfigArgs] = None,
        backfill_source: Optional[FeatureEngineeringKafkaConfigBackfillSourceArgs] = None,
        bootstrap_servers: Optional[str] = None,
        extra_options: Optional[Mapping[str, str]] = None,
        ingestion_config: Optional[FeatureEngineeringKafkaConfigIngestionConfigArgs] = None,
        key_schema: Optional[FeatureEngineeringKafkaConfigKeySchemaArgs] = None,
        name: Optional[str] = None,
        provider_config: Optional[FeatureEngineeringKafkaConfigProviderConfigArgs] = None,
        subscription_mode: Optional[FeatureEngineeringKafkaConfigSubscriptionModeArgs] = None,
        value_schema: Optional[FeatureEngineeringKafkaConfigValueSchemaArgs] = None) -> FeatureEngineeringKafkaConfig
    func GetFeatureEngineeringKafkaConfig(ctx *Context, name string, id IDInput, state *FeatureEngineeringKafkaConfigState, opts ...ResourceOption) (*FeatureEngineeringKafkaConfig, error)
    public static FeatureEngineeringKafkaConfig Get(string name, Input<string> id, FeatureEngineeringKafkaConfigState? state, CustomResourceOptions? opts = null)
    public static FeatureEngineeringKafkaConfig get(String name, Output<String> id, FeatureEngineeringKafkaConfigState state, CustomResourceOptions options)
    resources:  _:    type: databricks:FeatureEngineeringKafkaConfig    get:      id: ${id}
    import {
  to = databricks_featureengineeringkafkaconfig.example
  id = "${id}"
}
    name
    The unique name of the resulting resource.
    id
    The unique provider ID of the resource to lookup.
    state
    Any extra arguments used during the lookup.
    opts
    A bag of options that control this resource's behavior.
    resource_name
    The unique name of the resulting resource.
    id
    The unique provider ID of the resource to lookup.
    name
    The unique name of the resulting resource.
    id
    The unique provider ID of the resource to lookup.
    state
    Any extra arguments used during the lookup.
    opts
    A bag of options that control this resource's behavior.
    name
    The unique name of the resulting resource.
    id
    The unique provider ID of the resource to lookup.
    state
    Any extra arguments used during the lookup.
    opts
    A bag of options that control this resource's behavior.
    name
    The unique name of the resulting resource.
    id
    The unique provider ID of the resource to lookup.
    state
    Any extra arguments used during the lookup.
    opts
    A bag of options that control this resource's behavior.
    The following state arguments are supported:
    AuthConfig FeatureEngineeringKafkaConfigAuthConfig
    Authentication configuration for connection to topics
    BackfillSource FeatureEngineeringKafkaConfigBackfillSource
    A user-provided and managed source for backfilling data. Historical data is used when creating a training set from streaming features linked to this Kafka config. In the future, a separate table will be maintained by Databricks for forward filling data. The schema for this source must match exactly that of the key and value schemas specified for this Kafka config
    BootstrapServers string
    A comma-separated list of host/port pairs pointing to Kafka cluster
    ExtraOptions Dictionary<string, string>
    Catch-all for miscellaneous options. Keys should be source options or Kafka consumer options (kafka.*)
    IngestionConfig FeatureEngineeringKafkaConfigIngestionConfig
    Configuration for ingesting Kafka data into a Databricks-managed Delta table
    KeySchema FeatureEngineeringKafkaConfigKeySchema
    Schema configuration for extracting message keys from topics. At least one of keySchema and valueSchema must be provided
    Name string
    (string) - Name that uniquely identifies this Kafka config within the metastore. This will be the identifier used from the Feature object to reference these configs for a feature. Can be distinct from topic name
    ProviderConfig FeatureEngineeringKafkaConfigProviderConfig
    Configure the provider for management through account provider.
    SubscriptionMode FeatureEngineeringKafkaConfigSubscriptionMode
    Options to configure which Kafka topics to pull data from
    ValueSchema FeatureEngineeringKafkaConfigValueSchema
    Schema configuration for extracting message values from topics. At least one of keySchema and valueSchema must be provided
    AuthConfig FeatureEngineeringKafkaConfigAuthConfigArgs
    Authentication configuration for connection to topics
    BackfillSource FeatureEngineeringKafkaConfigBackfillSourceArgs
    A user-provided and managed source for backfilling data. Historical data is used when creating a training set from streaming features linked to this Kafka config. In the future, a separate table will be maintained by Databricks for forward filling data. The schema for this source must match exactly that of the key and value schemas specified for this Kafka config
    BootstrapServers string
    A comma-separated list of host/port pairs pointing to Kafka cluster
    ExtraOptions map[string]string
    Catch-all for miscellaneous options. Keys should be source options or Kafka consumer options (kafka.*)
    IngestionConfig FeatureEngineeringKafkaConfigIngestionConfigArgs
    Configuration for ingesting Kafka data into a Databricks-managed Delta table
    KeySchema FeatureEngineeringKafkaConfigKeySchemaArgs
    Schema configuration for extracting message keys from topics. At least one of keySchema and valueSchema must be provided
    Name string
    (string) - Name that uniquely identifies this Kafka config within the metastore. This will be the identifier used from the Feature object to reference these configs for a feature. Can be distinct from topic name
    ProviderConfig FeatureEngineeringKafkaConfigProviderConfigArgs
    Configure the provider for management through account provider.
    SubscriptionMode FeatureEngineeringKafkaConfigSubscriptionModeArgs
    Options to configure which Kafka topics to pull data from
    ValueSchema FeatureEngineeringKafkaConfigValueSchemaArgs
    Schema configuration for extracting message values from topics. At least one of keySchema and valueSchema must be provided
    auth_config object
    Authentication configuration for connection to topics
    backfill_source object
    A user-provided and managed source for backfilling data. Historical data is used when creating a training set from streaming features linked to this Kafka config. In the future, a separate table will be maintained by Databricks for forward filling data. The schema for this source must match exactly that of the key and value schemas specified for this Kafka config
    bootstrap_servers string
    A comma-separated list of host/port pairs pointing to Kafka cluster
    extra_options map(string)
    Catch-all for miscellaneous options. Keys should be source options or Kafka consumer options (kafka.*)
    ingestion_config object
    Configuration for ingesting Kafka data into a Databricks-managed Delta table
    key_schema object
    Schema configuration for extracting message keys from topics. At least one of keySchema and valueSchema must be provided
    name string
    (string) - Name that uniquely identifies this Kafka config within the metastore. This will be the identifier used from the Feature object to reference these configs for a feature. Can be distinct from topic name
    provider_config object
    Configure the provider for management through account provider.
    subscription_mode object
    Options to configure which Kafka topics to pull data from
    value_schema object
    Schema configuration for extracting message values from topics. At least one of keySchema and valueSchema must be provided
    authConfig FeatureEngineeringKafkaConfigAuthConfig
    Authentication configuration for connection to topics
    backfillSource FeatureEngineeringKafkaConfigBackfillSource
    A user-provided and managed source for backfilling data. Historical data is used when creating a training set from streaming features linked to this Kafka config. In the future, a separate table will be maintained by Databricks for forward filling data. The schema for this source must match exactly that of the key and value schemas specified for this Kafka config
    bootstrapServers String
    A comma-separated list of host/port pairs pointing to Kafka cluster
    extraOptions Map<String,String>
    Catch-all for miscellaneous options. Keys should be source options or Kafka consumer options (kafka.*)
    ingestionConfig FeatureEngineeringKafkaConfigIngestionConfig
    Configuration for ingesting Kafka data into a Databricks-managed Delta table
    keySchema FeatureEngineeringKafkaConfigKeySchema
    Schema configuration for extracting message keys from topics. At least one of keySchema and valueSchema must be provided
    name String
    (string) - Name that uniquely identifies this Kafka config within the metastore. This will be the identifier used from the Feature object to reference these configs for a feature. Can be distinct from topic name
    providerConfig FeatureEngineeringKafkaConfigProviderConfig
    Configure the provider for management through account provider.
    subscriptionMode FeatureEngineeringKafkaConfigSubscriptionMode
    Options to configure which Kafka topics to pull data from
    valueSchema FeatureEngineeringKafkaConfigValueSchema
    Schema configuration for extracting message values from topics. At least one of keySchema and valueSchema must be provided
    authConfig FeatureEngineeringKafkaConfigAuthConfig
    Authentication configuration for connection to topics
    backfillSource FeatureEngineeringKafkaConfigBackfillSource
    A user-provided and managed source for backfilling data. Historical data is used when creating a training set from streaming features linked to this Kafka config. In the future, a separate table will be maintained by Databricks for forward filling data. The schema for this source must match exactly that of the key and value schemas specified for this Kafka config
    bootstrapServers string
    A comma-separated list of host/port pairs pointing to Kafka cluster
    extraOptions {[key: string]: string}
    Catch-all for miscellaneous options. Keys should be source options or Kafka consumer options (kafka.*)
    ingestionConfig FeatureEngineeringKafkaConfigIngestionConfig
    Configuration for ingesting Kafka data into a Databricks-managed Delta table
    keySchema FeatureEngineeringKafkaConfigKeySchema
    Schema configuration for extracting message keys from topics. At least one of keySchema and valueSchema must be provided
    name string
    (string) - Name that uniquely identifies this Kafka config within the metastore. This will be the identifier used from the Feature object to reference these configs for a feature. Can be distinct from topic name
    providerConfig FeatureEngineeringKafkaConfigProviderConfig
    Configure the provider for management through account provider.
    subscriptionMode FeatureEngineeringKafkaConfigSubscriptionMode
    Options to configure which Kafka topics to pull data from
    valueSchema FeatureEngineeringKafkaConfigValueSchema
    Schema configuration for extracting message values from topics. At least one of keySchema and valueSchema must be provided
    auth_config FeatureEngineeringKafkaConfigAuthConfigArgs
    Authentication configuration for connection to topics
    backfill_source FeatureEngineeringKafkaConfigBackfillSourceArgs
    A user-provided and managed source for backfilling data. Historical data is used when creating a training set from streaming features linked to this Kafka config. In the future, a separate table will be maintained by Databricks for forward filling data. The schema for this source must match exactly that of the key and value schemas specified for this Kafka config
    bootstrap_servers str
    A comma-separated list of host/port pairs pointing to Kafka cluster
    extra_options Mapping[str, str]
    Catch-all for miscellaneous options. Keys should be source options or Kafka consumer options (kafka.*)
    ingestion_config FeatureEngineeringKafkaConfigIngestionConfigArgs
    Configuration for ingesting Kafka data into a Databricks-managed Delta table
    key_schema FeatureEngineeringKafkaConfigKeySchemaArgs
    Schema configuration for extracting message keys from topics. At least one of keySchema and valueSchema must be provided
    name str
    (string) - Name that uniquely identifies this Kafka config within the metastore. This will be the identifier used from the Feature object to reference these configs for a feature. Can be distinct from topic name
    provider_config FeatureEngineeringKafkaConfigProviderConfigArgs
    Configure the provider for management through account provider.
    subscription_mode FeatureEngineeringKafkaConfigSubscriptionModeArgs
    Options to configure which Kafka topics to pull data from
    value_schema FeatureEngineeringKafkaConfigValueSchemaArgs
    Schema configuration for extracting message values from topics. At least one of keySchema and valueSchema must be provided
    authConfig Property Map
    Authentication configuration for connection to topics
    backfillSource Property Map
    A user-provided and managed source for backfilling data. Historical data is used when creating a training set from streaming features linked to this Kafka config. In the future, a separate table will be maintained by Databricks for forward filling data. The schema for this source must match exactly that of the key and value schemas specified for this Kafka config
    bootstrapServers String
    A comma-separated list of host/port pairs pointing to Kafka cluster
    extraOptions Map<String>
    Catch-all for miscellaneous options. Keys should be source options or Kafka consumer options (kafka.*)
    ingestionConfig Property Map
    Configuration for ingesting Kafka data into a Databricks-managed Delta table
    keySchema Property Map
    Schema configuration for extracting message keys from topics. At least one of keySchema and valueSchema must be provided
    name String
    (string) - Name that uniquely identifies this Kafka config within the metastore. This will be the identifier used from the Feature object to reference these configs for a feature. Can be distinct from topic name
    providerConfig Property Map
    Configure the provider for management through account provider.
    subscriptionMode Property Map
    Options to configure which Kafka topics to pull data from
    valueSchema Property Map
    Schema configuration for extracting message values from topics. At least one of keySchema and valueSchema must be provided

    Supporting Types

    FeatureEngineeringKafkaConfigAuthConfig, FeatureEngineeringKafkaConfigAuthConfigArgs

    MtlsConfig FeatureEngineeringKafkaConfigAuthConfigMtlsConfig
    Mutual-TLS authentication. See MtlsConfig
    UcServiceCredentialName string
    Name of the Unity Catalog service credential. This value will be set under the option databricks.serviceCredential
    MtlsConfig FeatureEngineeringKafkaConfigAuthConfigMtlsConfig
    Mutual-TLS authentication. See MtlsConfig
    UcServiceCredentialName string
    Name of the Unity Catalog service credential. This value will be set under the option databricks.serviceCredential
    mtls_config object
    Mutual-TLS authentication. See MtlsConfig
    uc_service_credential_name string
    Name of the Unity Catalog service credential. This value will be set under the option databricks.serviceCredential
    mtlsConfig FeatureEngineeringKafkaConfigAuthConfigMtlsConfig
    Mutual-TLS authentication. See MtlsConfig
    ucServiceCredentialName String
    Name of the Unity Catalog service credential. This value will be set under the option databricks.serviceCredential
    mtlsConfig FeatureEngineeringKafkaConfigAuthConfigMtlsConfig
    Mutual-TLS authentication. See MtlsConfig
    ucServiceCredentialName string
    Name of the Unity Catalog service credential. This value will be set under the option databricks.serviceCredential
    mtls_config FeatureEngineeringKafkaConfigAuthConfigMtlsConfig
    Mutual-TLS authentication. See MtlsConfig
    uc_service_credential_name str
    Name of the Unity Catalog service credential. This value will be set under the option databricks.serviceCredential
    mtlsConfig Property Map
    Mutual-TLS authentication. See MtlsConfig
    ucServiceCredentialName String
    Name of the Unity Catalog service credential. This value will be set under the option databricks.serviceCredential

    FeatureEngineeringKafkaConfigAuthConfigMtlsConfig, FeatureEngineeringKafkaConfigAuthConfigMtlsConfigArgs

    KeyPasswordRef FeatureEngineeringKafkaConfigAuthConfigMtlsConfigKeyPasswordRef
    Secret-scope reference for the private key password. Often the same value as the keystore password (keytool's default), but provided as a separate field because Apache Kafka requires it as a distinct option (kafka.ssl.key.password)
    KeystoreLocation string
    Unity Catalog volume path to the JKS keystore file containing the client certificate and private key. e.g. "/Volumes////client.jks". The materialization compute must have read permission on this volume
    KeystorePasswordRef FeatureEngineeringKafkaConfigAuthConfigMtlsConfigKeystorePasswordRef
    Secret-scope reference for the JKS keystore password
    TruststoreLocation string
    Unity Catalog volume path to the JKS truststore file containing the CA certificate(s) trusted to verify the Kafka broker's server certificate. e.g. "/Volumes////truststore.jks"
    TruststorePasswordRef FeatureEngineeringKafkaConfigAuthConfigMtlsConfigTruststorePasswordRef
    Secret-scope reference for the JKS truststore password
    DisableHostnameVerification bool

    Set to true only when the broker certificate's SAN intentionally does not match the connection endpoint — for example when reaching the cluster through a PrivateLink endpoint whose DNS name is not in the broker certificate. Skipping the hostname check removes a defense against man-in-the-middle attacks; do not enable casually. mTLS client authentication is unaffected by this option.

    See the Apache Kafka SSL security guide for background on this check: https://kafka.apache.org/42/security/encryption-and-authentication-using-ssl/#host-name-verification

    KeyPasswordRef FeatureEngineeringKafkaConfigAuthConfigMtlsConfigKeyPasswordRef
    Secret-scope reference for the private key password. Often the same value as the keystore password (keytool's default), but provided as a separate field because Apache Kafka requires it as a distinct option (kafka.ssl.key.password)
    KeystoreLocation string
    Unity Catalog volume path to the JKS keystore file containing the client certificate and private key. e.g. "/Volumes////client.jks". The materialization compute must have read permission on this volume
    KeystorePasswordRef FeatureEngineeringKafkaConfigAuthConfigMtlsConfigKeystorePasswordRef
    Secret-scope reference for the JKS keystore password
    TruststoreLocation string
    Unity Catalog volume path to the JKS truststore file containing the CA certificate(s) trusted to verify the Kafka broker's server certificate. e.g. "/Volumes////truststore.jks"
    TruststorePasswordRef FeatureEngineeringKafkaConfigAuthConfigMtlsConfigTruststorePasswordRef
    Secret-scope reference for the JKS truststore password
    DisableHostnameVerification bool

    Set to true only when the broker certificate's SAN intentionally does not match the connection endpoint — for example when reaching the cluster through a PrivateLink endpoint whose DNS name is not in the broker certificate. Skipping the hostname check removes a defense against man-in-the-middle attacks; do not enable casually. mTLS client authentication is unaffected by this option.

    See the Apache Kafka SSL security guide for background on this check: https://kafka.apache.org/42/security/encryption-and-authentication-using-ssl/#host-name-verification

    key_password_ref object
    Secret-scope reference for the private key password. Often the same value as the keystore password (keytool's default), but provided as a separate field because Apache Kafka requires it as a distinct option (kafka.ssl.key.password)
    keystore_location string
    Unity Catalog volume path to the JKS keystore file containing the client certificate and private key. e.g. "/Volumes////client.jks". The materialization compute must have read permission on this volume
    keystore_password_ref object
    Secret-scope reference for the JKS keystore password
    truststore_location string
    Unity Catalog volume path to the JKS truststore file containing the CA certificate(s) trusted to verify the Kafka broker's server certificate. e.g. "/Volumes////truststore.jks"
    truststore_password_ref object
    Secret-scope reference for the JKS truststore password
    disable_hostname_verification bool

    Set to true only when the broker certificate's SAN intentionally does not match the connection endpoint — for example when reaching the cluster through a PrivateLink endpoint whose DNS name is not in the broker certificate. Skipping the hostname check removes a defense against man-in-the-middle attacks; do not enable casually. mTLS client authentication is unaffected by this option.

    See the Apache Kafka SSL security guide for background on this check: https://kafka.apache.org/42/security/encryption-and-authentication-using-ssl/#host-name-verification

    keyPasswordRef FeatureEngineeringKafkaConfigAuthConfigMtlsConfigKeyPasswordRef
    Secret-scope reference for the private key password. Often the same value as the keystore password (keytool's default), but provided as a separate field because Apache Kafka requires it as a distinct option (kafka.ssl.key.password)
    keystoreLocation String
    Unity Catalog volume path to the JKS keystore file containing the client certificate and private key. e.g. "/Volumes////client.jks". The materialization compute must have read permission on this volume
    keystorePasswordRef FeatureEngineeringKafkaConfigAuthConfigMtlsConfigKeystorePasswordRef
    Secret-scope reference for the JKS keystore password
    truststoreLocation String
    Unity Catalog volume path to the JKS truststore file containing the CA certificate(s) trusted to verify the Kafka broker's server certificate. e.g. "/Volumes////truststore.jks"
    truststorePasswordRef FeatureEngineeringKafkaConfigAuthConfigMtlsConfigTruststorePasswordRef
    Secret-scope reference for the JKS truststore password
    disableHostnameVerification Boolean

    Set to true only when the broker certificate's SAN intentionally does not match the connection endpoint — for example when reaching the cluster through a PrivateLink endpoint whose DNS name is not in the broker certificate. Skipping the hostname check removes a defense against man-in-the-middle attacks; do not enable casually. mTLS client authentication is unaffected by this option.

    See the Apache Kafka SSL security guide for background on this check: https://kafka.apache.org/42/security/encryption-and-authentication-using-ssl/#host-name-verification

    keyPasswordRef FeatureEngineeringKafkaConfigAuthConfigMtlsConfigKeyPasswordRef
    Secret-scope reference for the private key password. Often the same value as the keystore password (keytool's default), but provided as a separate field because Apache Kafka requires it as a distinct option (kafka.ssl.key.password)
    keystoreLocation string
    Unity Catalog volume path to the JKS keystore file containing the client certificate and private key. e.g. "/Volumes////client.jks". The materialization compute must have read permission on this volume
    keystorePasswordRef FeatureEngineeringKafkaConfigAuthConfigMtlsConfigKeystorePasswordRef
    Secret-scope reference for the JKS keystore password
    truststoreLocation string
    Unity Catalog volume path to the JKS truststore file containing the CA certificate(s) trusted to verify the Kafka broker's server certificate. e.g. "/Volumes////truststore.jks"
    truststorePasswordRef FeatureEngineeringKafkaConfigAuthConfigMtlsConfigTruststorePasswordRef
    Secret-scope reference for the JKS truststore password
    disableHostnameVerification boolean

    Set to true only when the broker certificate's SAN intentionally does not match the connection endpoint — for example when reaching the cluster through a PrivateLink endpoint whose DNS name is not in the broker certificate. Skipping the hostname check removes a defense against man-in-the-middle attacks; do not enable casually. mTLS client authentication is unaffected by this option.

    See the Apache Kafka SSL security guide for background on this check: https://kafka.apache.org/42/security/encryption-and-authentication-using-ssl/#host-name-verification

    key_password_ref FeatureEngineeringKafkaConfigAuthConfigMtlsConfigKeyPasswordRef
    Secret-scope reference for the private key password. Often the same value as the keystore password (keytool's default), but provided as a separate field because Apache Kafka requires it as a distinct option (kafka.ssl.key.password)
    keystore_location str
    Unity Catalog volume path to the JKS keystore file containing the client certificate and private key. e.g. "/Volumes////client.jks". The materialization compute must have read permission on this volume
    keystore_password_ref FeatureEngineeringKafkaConfigAuthConfigMtlsConfigKeystorePasswordRef
    Secret-scope reference for the JKS keystore password
    truststore_location str
    Unity Catalog volume path to the JKS truststore file containing the CA certificate(s) trusted to verify the Kafka broker's server certificate. e.g. "/Volumes////truststore.jks"
    truststore_password_ref FeatureEngineeringKafkaConfigAuthConfigMtlsConfigTruststorePasswordRef
    Secret-scope reference for the JKS truststore password
    disable_hostname_verification bool

    Set to true only when the broker certificate's SAN intentionally does not match the connection endpoint — for example when reaching the cluster through a PrivateLink endpoint whose DNS name is not in the broker certificate. Skipping the hostname check removes a defense against man-in-the-middle attacks; do not enable casually. mTLS client authentication is unaffected by this option.

    See the Apache Kafka SSL security guide for background on this check: https://kafka.apache.org/42/security/encryption-and-authentication-using-ssl/#host-name-verification

    keyPasswordRef Property Map
    Secret-scope reference for the private key password. Often the same value as the keystore password (keytool's default), but provided as a separate field because Apache Kafka requires it as a distinct option (kafka.ssl.key.password)
    keystoreLocation String
    Unity Catalog volume path to the JKS keystore file containing the client certificate and private key. e.g. "/Volumes////client.jks". The materialization compute must have read permission on this volume
    keystorePasswordRef Property Map
    Secret-scope reference for the JKS keystore password
    truststoreLocation String
    Unity Catalog volume path to the JKS truststore file containing the CA certificate(s) trusted to verify the Kafka broker's server certificate. e.g. "/Volumes////truststore.jks"
    truststorePasswordRef Property Map
    Secret-scope reference for the JKS truststore password
    disableHostnameVerification Boolean

    Set to true only when the broker certificate's SAN intentionally does not match the connection endpoint — for example when reaching the cluster through a PrivateLink endpoint whose DNS name is not in the broker certificate. Skipping the hostname check removes a defense against man-in-the-middle attacks; do not enable casually. mTLS client authentication is unaffected by this option.

    See the Apache Kafka SSL security guide for background on this check: https://kafka.apache.org/42/security/encryption-and-authentication-using-ssl/#host-name-verification

    FeatureEngineeringKafkaConfigAuthConfigMtlsConfigKeyPasswordRef, FeatureEngineeringKafkaConfigAuthConfigMtlsConfigKeyPasswordRefArgs

    Key string
    The key within the scope
    Scope string
    The Databricks secret scope name
    Key string
    The key within the scope
    Scope string
    The Databricks secret scope name
    key string
    The key within the scope
    scope string
    The Databricks secret scope name
    key String
    The key within the scope
    scope String
    The Databricks secret scope name
    key string
    The key within the scope
    scope string
    The Databricks secret scope name
    key str
    The key within the scope
    scope str
    The Databricks secret scope name
    key String
    The key within the scope
    scope String
    The Databricks secret scope name

    FeatureEngineeringKafkaConfigAuthConfigMtlsConfigKeystorePasswordRef, FeatureEngineeringKafkaConfigAuthConfigMtlsConfigKeystorePasswordRefArgs

    Key string
    The key within the scope
    Scope string
    The Databricks secret scope name
    Key string
    The key within the scope
    Scope string
    The Databricks secret scope name
    key string
    The key within the scope
    scope string
    The Databricks secret scope name
    key String
    The key within the scope
    scope String
    The Databricks secret scope name
    key string
    The key within the scope
    scope string
    The Databricks secret scope name
    key str
    The key within the scope
    scope str
    The Databricks secret scope name
    key String
    The key within the scope
    scope String
    The Databricks secret scope name

    FeatureEngineeringKafkaConfigAuthConfigMtlsConfigTruststorePasswordRef, FeatureEngineeringKafkaConfigAuthConfigMtlsConfigTruststorePasswordRefArgs

    Key string
    The key within the scope
    Scope string
    The Databricks secret scope name
    Key string
    The key within the scope
    Scope string
    The Databricks secret scope name
    key string
    The key within the scope
    scope string
    The Databricks secret scope name
    key String
    The key within the scope
    scope String
    The Databricks secret scope name
    key string
    The key within the scope
    scope string
    The Databricks secret scope name
    key str
    The key within the scope
    scope str
    The Databricks secret scope name
    key String
    The key within the scope
    scope String
    The Databricks secret scope name

    FeatureEngineeringKafkaConfigBackfillSource, FeatureEngineeringKafkaConfigBackfillSourceArgs

    DeltaTableName string
    DeltaTableSource FeatureEngineeringKafkaConfigBackfillSourceDeltaTableSource
    Deprecated: Use deltaTableName instead. Kept for backwards compatibility. The Delta table source containing the historical data to backfill. Only the delta table name is used for backfill, other fields are ignored
    DeltaTableName string
    DeltaTableSource FeatureEngineeringKafkaConfigBackfillSourceDeltaTableSource
    Deprecated: Use deltaTableName instead. Kept for backwards compatibility. The Delta table source containing the historical data to backfill. Only the delta table name is used for backfill, other fields are ignored
    delta_table_name string
    delta_table_source object
    Deprecated: Use deltaTableName instead. Kept for backwards compatibility. The Delta table source containing the historical data to backfill. Only the delta table name is used for backfill, other fields are ignored
    deltaTableName String
    deltaTableSource FeatureEngineeringKafkaConfigBackfillSourceDeltaTableSource
    Deprecated: Use deltaTableName instead. Kept for backwards compatibility. The Delta table source containing the historical data to backfill. Only the delta table name is used for backfill, other fields are ignored
    deltaTableName string
    deltaTableSource FeatureEngineeringKafkaConfigBackfillSourceDeltaTableSource
    Deprecated: Use deltaTableName instead. Kept for backwards compatibility. The Delta table source containing the historical data to backfill. Only the delta table name is used for backfill, other fields are ignored
    delta_table_name str
    delta_table_source FeatureEngineeringKafkaConfigBackfillSourceDeltaTableSource
    Deprecated: Use deltaTableName instead. Kept for backwards compatibility. The Delta table source containing the historical data to backfill. Only the delta table name is used for backfill, other fields are ignored
    deltaTableName String
    deltaTableSource Property Map
    Deprecated: Use deltaTableName instead. Kept for backwards compatibility. The Delta table source containing the historical data to backfill. Only the delta table name is used for backfill, other fields are ignored

    FeatureEngineeringKafkaConfigBackfillSourceDeltaTableSource, FeatureEngineeringKafkaConfigBackfillSourceDeltaTableSourceArgs

    FullName string
    The full three-part (catalog, schema, table) name of the Delta table
    DataframeSchema string
    Schema of the resulting dataframe after transformations, in Spark StructType JSON format (from df.schema.json()). Required if transformationSql is specified. Example: {"type":"struct","fields":[{"name":"colA","type":"integer","nullable":true,"metadata":{}},{"name":"colC","type":"integer","nullable":true,"metadata":{}}]}
    EntityColumns List<string>
    Deprecated: Use Feature.entity instead. Kept for backwards compatibility. The entity columns of the Delta table
    FilterCondition string
    Single WHERE clause to filter delta table before applying transformations. Will be row-wise evaluated, so should only include conditionals and projections
    TimeseriesColumn string
    Deprecated: Use Feature.timeseries_column instead. Kept for backwards compatibility. The timeseries column of the Delta table
    TransformationSql string
    A single SQL SELECT expression applied after filter_condition. Should contains all the columns needed (eg. "SELECT , colA + colB AS colC FROM x.y.z WHERE colA > 0" would have transformationSql ", colA + colB AS colC") If transformationSql is not provided, all columns of the delta table are present in the DataSource dataframe
    FullName string
    The full three-part (catalog, schema, table) name of the Delta table
    DataframeSchema string
    Schema of the resulting dataframe after transformations, in Spark StructType JSON format (from df.schema.json()). Required if transformationSql is specified. Example: {"type":"struct","fields":[{"name":"colA","type":"integer","nullable":true,"metadata":{}},{"name":"colC","type":"integer","nullable":true,"metadata":{}}]}
    EntityColumns []string
    Deprecated: Use Feature.entity instead. Kept for backwards compatibility. The entity columns of the Delta table
    FilterCondition string
    Single WHERE clause to filter delta table before applying transformations. Will be row-wise evaluated, so should only include conditionals and projections
    TimeseriesColumn string
    Deprecated: Use Feature.timeseries_column instead. Kept for backwards compatibility. The timeseries column of the Delta table
    TransformationSql string
    A single SQL SELECT expression applied after filter_condition. Should contains all the columns needed (eg. "SELECT , colA + colB AS colC FROM x.y.z WHERE colA > 0" would have transformationSql ", colA + colB AS colC") If transformationSql is not provided, all columns of the delta table are present in the DataSource dataframe
    full_name string
    The full three-part (catalog, schema, table) name of the Delta table
    dataframe_schema string
    Schema of the resulting dataframe after transformations, in Spark StructType JSON format (from df.schema.json()). Required if transformationSql is specified. Example: {"type":"struct","fields":[{"name":"colA","type":"integer","nullable":true,"metadata":{}},{"name":"colC","type":"integer","nullable":true,"metadata":{}}]}
    entity_columns list(string)
    Deprecated: Use Feature.entity instead. Kept for backwards compatibility. The entity columns of the Delta table
    filter_condition string
    Single WHERE clause to filter delta table before applying transformations. Will be row-wise evaluated, so should only include conditionals and projections
    timeseries_column string
    Deprecated: Use Feature.timeseries_column instead. Kept for backwards compatibility. The timeseries column of the Delta table
    transformation_sql string
    A single SQL SELECT expression applied after filter_condition. Should contains all the columns needed (eg. "SELECT , colA + colB AS colC FROM x.y.z WHERE colA > 0" would have transformationSql ", colA + colB AS colC") If transformationSql is not provided, all columns of the delta table are present in the DataSource dataframe
    fullName String
    The full three-part (catalog, schema, table) name of the Delta table
    dataframeSchema String
    Schema of the resulting dataframe after transformations, in Spark StructType JSON format (from df.schema.json()). Required if transformationSql is specified. Example: {"type":"struct","fields":[{"name":"colA","type":"integer","nullable":true,"metadata":{}},{"name":"colC","type":"integer","nullable":true,"metadata":{}}]}
    entityColumns List<String>
    Deprecated: Use Feature.entity instead. Kept for backwards compatibility. The entity columns of the Delta table
    filterCondition String
    Single WHERE clause to filter delta table before applying transformations. Will be row-wise evaluated, so should only include conditionals and projections
    timeseriesColumn String
    Deprecated: Use Feature.timeseries_column instead. Kept for backwards compatibility. The timeseries column of the Delta table
    transformationSql String
    A single SQL SELECT expression applied after filter_condition. Should contains all the columns needed (eg. "SELECT , colA + colB AS colC FROM x.y.z WHERE colA > 0" would have transformationSql ", colA + colB AS colC") If transformationSql is not provided, all columns of the delta table are present in the DataSource dataframe
    fullName string
    The full three-part (catalog, schema, table) name of the Delta table
    dataframeSchema string
    Schema of the resulting dataframe after transformations, in Spark StructType JSON format (from df.schema.json()). Required if transformationSql is specified. Example: {"type":"struct","fields":[{"name":"colA","type":"integer","nullable":true,"metadata":{}},{"name":"colC","type":"integer","nullable":true,"metadata":{}}]}
    entityColumns string[]
    Deprecated: Use Feature.entity instead. Kept for backwards compatibility. The entity columns of the Delta table
    filterCondition string
    Single WHERE clause to filter delta table before applying transformations. Will be row-wise evaluated, so should only include conditionals and projections
    timeseriesColumn string
    Deprecated: Use Feature.timeseries_column instead. Kept for backwards compatibility. The timeseries column of the Delta table
    transformationSql string
    A single SQL SELECT expression applied after filter_condition. Should contains all the columns needed (eg. "SELECT , colA + colB AS colC FROM x.y.z WHERE colA > 0" would have transformationSql ", colA + colB AS colC") If transformationSql is not provided, all columns of the delta table are present in the DataSource dataframe
    full_name str
    The full three-part (catalog, schema, table) name of the Delta table
    dataframe_schema str
    Schema of the resulting dataframe after transformations, in Spark StructType JSON format (from df.schema.json()). Required if transformationSql is specified. Example: {"type":"struct","fields":[{"name":"colA","type":"integer","nullable":true,"metadata":{}},{"name":"colC","type":"integer","nullable":true,"metadata":{}}]}
    entity_columns Sequence[str]
    Deprecated: Use Feature.entity instead. Kept for backwards compatibility. The entity columns of the Delta table
    filter_condition str
    Single WHERE clause to filter delta table before applying transformations. Will be row-wise evaluated, so should only include conditionals and projections
    timeseries_column str
    Deprecated: Use Feature.timeseries_column instead. Kept for backwards compatibility. The timeseries column of the Delta table
    transformation_sql str
    A single SQL SELECT expression applied after filter_condition. Should contains all the columns needed (eg. "SELECT , colA + colB AS colC FROM x.y.z WHERE colA > 0" would have transformationSql ", colA + colB AS colC") If transformationSql is not provided, all columns of the delta table are present in the DataSource dataframe
    fullName String
    The full three-part (catalog, schema, table) name of the Delta table
    dataframeSchema String
    Schema of the resulting dataframe after transformations, in Spark StructType JSON format (from df.schema.json()). Required if transformationSql is specified. Example: {"type":"struct","fields":[{"name":"colA","type":"integer","nullable":true,"metadata":{}},{"name":"colC","type":"integer","nullable":true,"metadata":{}}]}
    entityColumns List<String>
    Deprecated: Use Feature.entity instead. Kept for backwards compatibility. The entity columns of the Delta table
    filterCondition String
    Single WHERE clause to filter delta table before applying transformations. Will be row-wise evaluated, so should only include conditionals and projections
    timeseriesColumn String
    Deprecated: Use Feature.timeseries_column instead. Kept for backwards compatibility. The timeseries column of the Delta table
    transformationSql String
    A single SQL SELECT expression applied after filter_condition. Should contains all the columns needed (eg. "SELECT , colA + colB AS colC FROM x.y.z WHERE colA > 0" would have transformationSql ", colA + colB AS colC") If transformationSql is not provided, all columns of the delta table are present in the DataSource dataframe

    FeatureEngineeringKafkaConfigIngestionConfig, FeatureEngineeringKafkaConfigIngestionConfigArgs

    IngestionDestination FeatureEngineeringKafkaConfigIngestionConfigIngestionDestination
    Destination for the Databricks-managed Delta table that holds an offline copy of the streaming data for querying and training. This table contains both 1) forward-filled data from the Stream and 2) backfilled data from the BackfillSource (if provided). This table is created and managed by Databricks and is deleted when the Stream is deleted
    BackfillJobId int
    (integer) - The ID of the Databricks Job that performs the historical backfill of the ingestion Delta table
    BackfillSource FeatureEngineeringKafkaConfigIngestionConfigBackfillSource
    A user-provided and managed source for backfilling data. Historical data is used when creating a training set from streaming features linked to this Kafka config. In the future, a separate table will be maintained by Databricks for forward filling data. The schema for this source must match exactly that of the key and value schemas specified for this Kafka config
    DeduplicationColumns List<string>
    Column paths used to identify duplicate rows during ingestion; only one row per distinct combination of these values is kept. Use dot notation for nested fields (e.g. value.user_id). Empty list means every column is compared
    IngestionJobId int
    (integer) - The ID of the Databricks Job that performs the forward-fill ingestion
    IngestionPipelineId string
    (string) - The ID of the SDP pipeline that continuously copies new events from the streaming source into the ingestion Delta table
    IngestionDestination FeatureEngineeringKafkaConfigIngestionConfigIngestionDestination
    Destination for the Databricks-managed Delta table that holds an offline copy of the streaming data for querying and training. This table contains both 1) forward-filled data from the Stream and 2) backfilled data from the BackfillSource (if provided). This table is created and managed by Databricks and is deleted when the Stream is deleted
    BackfillJobId int
    (integer) - The ID of the Databricks Job that performs the historical backfill of the ingestion Delta table
    BackfillSource FeatureEngineeringKafkaConfigIngestionConfigBackfillSource
    A user-provided and managed source for backfilling data. Historical data is used when creating a training set from streaming features linked to this Kafka config. In the future, a separate table will be maintained by Databricks for forward filling data. The schema for this source must match exactly that of the key and value schemas specified for this Kafka config
    DeduplicationColumns []string
    Column paths used to identify duplicate rows during ingestion; only one row per distinct combination of these values is kept. Use dot notation for nested fields (e.g. value.user_id). Empty list means every column is compared
    IngestionJobId int
    (integer) - The ID of the Databricks Job that performs the forward-fill ingestion
    IngestionPipelineId string
    (string) - The ID of the SDP pipeline that continuously copies new events from the streaming source into the ingestion Delta table
    ingestion_destination object
    Destination for the Databricks-managed Delta table that holds an offline copy of the streaming data for querying and training. This table contains both 1) forward-filled data from the Stream and 2) backfilled data from the BackfillSource (if provided). This table is created and managed by Databricks and is deleted when the Stream is deleted
    backfill_job_id number
    (integer) - The ID of the Databricks Job that performs the historical backfill of the ingestion Delta table
    backfill_source object
    A user-provided and managed source for backfilling data. Historical data is used when creating a training set from streaming features linked to this Kafka config. In the future, a separate table will be maintained by Databricks for forward filling data. The schema for this source must match exactly that of the key and value schemas specified for this Kafka config
    deduplication_columns list(string)
    Column paths used to identify duplicate rows during ingestion; only one row per distinct combination of these values is kept. Use dot notation for nested fields (e.g. value.user_id). Empty list means every column is compared
    ingestion_job_id number
    (integer) - The ID of the Databricks Job that performs the forward-fill ingestion
    ingestion_pipeline_id string
    (string) - The ID of the SDP pipeline that continuously copies new events from the streaming source into the ingestion Delta table
    ingestionDestination FeatureEngineeringKafkaConfigIngestionConfigIngestionDestination
    Destination for the Databricks-managed Delta table that holds an offline copy of the streaming data for querying and training. This table contains both 1) forward-filled data from the Stream and 2) backfilled data from the BackfillSource (if provided). This table is created and managed by Databricks and is deleted when the Stream is deleted
    backfillJobId Integer
    (integer) - The ID of the Databricks Job that performs the historical backfill of the ingestion Delta table
    backfillSource FeatureEngineeringKafkaConfigIngestionConfigBackfillSource
    A user-provided and managed source for backfilling data. Historical data is used when creating a training set from streaming features linked to this Kafka config. In the future, a separate table will be maintained by Databricks for forward filling data. The schema for this source must match exactly that of the key and value schemas specified for this Kafka config
    deduplicationColumns List<String>
    Column paths used to identify duplicate rows during ingestion; only one row per distinct combination of these values is kept. Use dot notation for nested fields (e.g. value.user_id). Empty list means every column is compared
    ingestionJobId Integer
    (integer) - The ID of the Databricks Job that performs the forward-fill ingestion
    ingestionPipelineId String
    (string) - The ID of the SDP pipeline that continuously copies new events from the streaming source into the ingestion Delta table
    ingestionDestination FeatureEngineeringKafkaConfigIngestionConfigIngestionDestination
    Destination for the Databricks-managed Delta table that holds an offline copy of the streaming data for querying and training. This table contains both 1) forward-filled data from the Stream and 2) backfilled data from the BackfillSource (if provided). This table is created and managed by Databricks and is deleted when the Stream is deleted
    backfillJobId number
    (integer) - The ID of the Databricks Job that performs the historical backfill of the ingestion Delta table
    backfillSource FeatureEngineeringKafkaConfigIngestionConfigBackfillSource
    A user-provided and managed source for backfilling data. Historical data is used when creating a training set from streaming features linked to this Kafka config. In the future, a separate table will be maintained by Databricks for forward filling data. The schema for this source must match exactly that of the key and value schemas specified for this Kafka config
    deduplicationColumns string[]
    Column paths used to identify duplicate rows during ingestion; only one row per distinct combination of these values is kept. Use dot notation for nested fields (e.g. value.user_id). Empty list means every column is compared
    ingestionJobId number
    (integer) - The ID of the Databricks Job that performs the forward-fill ingestion
    ingestionPipelineId string
    (string) - The ID of the SDP pipeline that continuously copies new events from the streaming source into the ingestion Delta table
    ingestion_destination FeatureEngineeringKafkaConfigIngestionConfigIngestionDestination
    Destination for the Databricks-managed Delta table that holds an offline copy of the streaming data for querying and training. This table contains both 1) forward-filled data from the Stream and 2) backfilled data from the BackfillSource (if provided). This table is created and managed by Databricks and is deleted when the Stream is deleted
    backfill_job_id int
    (integer) - The ID of the Databricks Job that performs the historical backfill of the ingestion Delta table
    backfill_source FeatureEngineeringKafkaConfigIngestionConfigBackfillSource
    A user-provided and managed source for backfilling data. Historical data is used when creating a training set from streaming features linked to this Kafka config. In the future, a separate table will be maintained by Databricks for forward filling data. The schema for this source must match exactly that of the key and value schemas specified for this Kafka config
    deduplication_columns Sequence[str]
    Column paths used to identify duplicate rows during ingestion; only one row per distinct combination of these values is kept. Use dot notation for nested fields (e.g. value.user_id). Empty list means every column is compared
    ingestion_job_id int
    (integer) - The ID of the Databricks Job that performs the forward-fill ingestion
    ingestion_pipeline_id str
    (string) - The ID of the SDP pipeline that continuously copies new events from the streaming source into the ingestion Delta table
    ingestionDestination Property Map
    Destination for the Databricks-managed Delta table that holds an offline copy of the streaming data for querying and training. This table contains both 1) forward-filled data from the Stream and 2) backfilled data from the BackfillSource (if provided). This table is created and managed by Databricks and is deleted when the Stream is deleted
    backfillJobId Number
    (integer) - The ID of the Databricks Job that performs the historical backfill of the ingestion Delta table
    backfillSource Property Map
    A user-provided and managed source for backfilling data. Historical data is used when creating a training set from streaming features linked to this Kafka config. In the future, a separate table will be maintained by Databricks for forward filling data. The schema for this source must match exactly that of the key and value schemas specified for this Kafka config
    deduplicationColumns List<String>
    Column paths used to identify duplicate rows during ingestion; only one row per distinct combination of these values is kept. Use dot notation for nested fields (e.g. value.user_id). Empty list means every column is compared
    ingestionJobId Number
    (integer) - The ID of the Databricks Job that performs the forward-fill ingestion
    ingestionPipelineId String
    (string) - The ID of the SDP pipeline that continuously copies new events from the streaming source into the ingestion Delta table

    FeatureEngineeringKafkaConfigIngestionConfigBackfillSource, FeatureEngineeringKafkaConfigIngestionConfigBackfillSourceArgs

    DeltaTableName string
    DeltaTableSource FeatureEngineeringKafkaConfigIngestionConfigBackfillSourceDeltaTableSource
    Deprecated: Use deltaTableName instead. Kept for backwards compatibility. The Delta table source containing the historical data to backfill. Only the delta table name is used for backfill, other fields are ignored
    DeltaTableName string
    DeltaTableSource FeatureEngineeringKafkaConfigIngestionConfigBackfillSourceDeltaTableSource
    Deprecated: Use deltaTableName instead. Kept for backwards compatibility. The Delta table source containing the historical data to backfill. Only the delta table name is used for backfill, other fields are ignored
    delta_table_name string
    delta_table_source object
    Deprecated: Use deltaTableName instead. Kept for backwards compatibility. The Delta table source containing the historical data to backfill. Only the delta table name is used for backfill, other fields are ignored
    deltaTableName String
    deltaTableSource FeatureEngineeringKafkaConfigIngestionConfigBackfillSourceDeltaTableSource
    Deprecated: Use deltaTableName instead. Kept for backwards compatibility. The Delta table source containing the historical data to backfill. Only the delta table name is used for backfill, other fields are ignored
    deltaTableName string
    deltaTableSource FeatureEngineeringKafkaConfigIngestionConfigBackfillSourceDeltaTableSource
    Deprecated: Use deltaTableName instead. Kept for backwards compatibility. The Delta table source containing the historical data to backfill. Only the delta table name is used for backfill, other fields are ignored
    delta_table_name str
    delta_table_source FeatureEngineeringKafkaConfigIngestionConfigBackfillSourceDeltaTableSource
    Deprecated: Use deltaTableName instead. Kept for backwards compatibility. The Delta table source containing the historical data to backfill. Only the delta table name is used for backfill, other fields are ignored
    deltaTableName String
    deltaTableSource Property Map
    Deprecated: Use deltaTableName instead. Kept for backwards compatibility. The Delta table source containing the historical data to backfill. Only the delta table name is used for backfill, other fields are ignored

    FeatureEngineeringKafkaConfigIngestionConfigBackfillSourceDeltaTableSource, FeatureEngineeringKafkaConfigIngestionConfigBackfillSourceDeltaTableSourceArgs

    FullName string
    The full three-part (catalog, schema, table) name of the Delta table
    DataframeSchema string
    Schema of the resulting dataframe after transformations, in Spark StructType JSON format (from df.schema.json()). Required if transformationSql is specified. Example: {"type":"struct","fields":[{"name":"colA","type":"integer","nullable":true,"metadata":{}},{"name":"colC","type":"integer","nullable":true,"metadata":{}}]}
    EntityColumns List<string>
    Deprecated: Use Feature.entity instead. Kept for backwards compatibility. The entity columns of the Delta table
    FilterCondition string
    Single WHERE clause to filter delta table before applying transformations. Will be row-wise evaluated, so should only include conditionals and projections
    TimeseriesColumn string
    Deprecated: Use Feature.timeseries_column instead. Kept for backwards compatibility. The timeseries column of the Delta table
    TransformationSql string
    A single SQL SELECT expression applied after filter_condition. Should contains all the columns needed (eg. "SELECT , colA + colB AS colC FROM x.y.z WHERE colA > 0" would have transformationSql ", colA + colB AS colC") If transformationSql is not provided, all columns of the delta table are present in the DataSource dataframe
    FullName string
    The full three-part (catalog, schema, table) name of the Delta table
    DataframeSchema string
    Schema of the resulting dataframe after transformations, in Spark StructType JSON format (from df.schema.json()). Required if transformationSql is specified. Example: {"type":"struct","fields":[{"name":"colA","type":"integer","nullable":true,"metadata":{}},{"name":"colC","type":"integer","nullable":true,"metadata":{}}]}
    EntityColumns []string
    Deprecated: Use Feature.entity instead. Kept for backwards compatibility. The entity columns of the Delta table
    FilterCondition string
    Single WHERE clause to filter delta table before applying transformations. Will be row-wise evaluated, so should only include conditionals and projections
    TimeseriesColumn string
    Deprecated: Use Feature.timeseries_column instead. Kept for backwards compatibility. The timeseries column of the Delta table
    TransformationSql string
    A single SQL SELECT expression applied after filter_condition. Should contains all the columns needed (eg. "SELECT , colA + colB AS colC FROM x.y.z WHERE colA > 0" would have transformationSql ", colA + colB AS colC") If transformationSql is not provided, all columns of the delta table are present in the DataSource dataframe
    full_name string
    The full three-part (catalog, schema, table) name of the Delta table
    dataframe_schema string
    Schema of the resulting dataframe after transformations, in Spark StructType JSON format (from df.schema.json()). Required if transformationSql is specified. Example: {"type":"struct","fields":[{"name":"colA","type":"integer","nullable":true,"metadata":{}},{"name":"colC","type":"integer","nullable":true,"metadata":{}}]}
    entity_columns list(string)
    Deprecated: Use Feature.entity instead. Kept for backwards compatibility. The entity columns of the Delta table
    filter_condition string
    Single WHERE clause to filter delta table before applying transformations. Will be row-wise evaluated, so should only include conditionals and projections
    timeseries_column string
    Deprecated: Use Feature.timeseries_column instead. Kept for backwards compatibility. The timeseries column of the Delta table
    transformation_sql string
    A single SQL SELECT expression applied after filter_condition. Should contains all the columns needed (eg. "SELECT , colA + colB AS colC FROM x.y.z WHERE colA > 0" would have transformationSql ", colA + colB AS colC") If transformationSql is not provided, all columns of the delta table are present in the DataSource dataframe
    fullName String
    The full three-part (catalog, schema, table) name of the Delta table
    dataframeSchema String
    Schema of the resulting dataframe after transformations, in Spark StructType JSON format (from df.schema.json()). Required if transformationSql is specified. Example: {"type":"struct","fields":[{"name":"colA","type":"integer","nullable":true,"metadata":{}},{"name":"colC","type":"integer","nullable":true,"metadata":{}}]}
    entityColumns List<String>
    Deprecated: Use Feature.entity instead. Kept for backwards compatibility. The entity columns of the Delta table
    filterCondition String
    Single WHERE clause to filter delta table before applying transformations. Will be row-wise evaluated, so should only include conditionals and projections
    timeseriesColumn String
    Deprecated: Use Feature.timeseries_column instead. Kept for backwards compatibility. The timeseries column of the Delta table
    transformationSql String
    A single SQL SELECT expression applied after filter_condition. Should contains all the columns needed (eg. "SELECT , colA + colB AS colC FROM x.y.z WHERE colA > 0" would have transformationSql ", colA + colB AS colC") If transformationSql is not provided, all columns of the delta table are present in the DataSource dataframe
    fullName string
    The full three-part (catalog, schema, table) name of the Delta table
    dataframeSchema string
    Schema of the resulting dataframe after transformations, in Spark StructType JSON format (from df.schema.json()). Required if transformationSql is specified. Example: {"type":"struct","fields":[{"name":"colA","type":"integer","nullable":true,"metadata":{}},{"name":"colC","type":"integer","nullable":true,"metadata":{}}]}
    entityColumns string[]
    Deprecated: Use Feature.entity instead. Kept for backwards compatibility. The entity columns of the Delta table
    filterCondition string
    Single WHERE clause to filter delta table before applying transformations. Will be row-wise evaluated, so should only include conditionals and projections
    timeseriesColumn string
    Deprecated: Use Feature.timeseries_column instead. Kept for backwards compatibility. The timeseries column of the Delta table
    transformationSql string
    A single SQL SELECT expression applied after filter_condition. Should contains all the columns needed (eg. "SELECT , colA + colB AS colC FROM x.y.z WHERE colA > 0" would have transformationSql ", colA + colB AS colC") If transformationSql is not provided, all columns of the delta table are present in the DataSource dataframe
    full_name str
    The full three-part (catalog, schema, table) name of the Delta table
    dataframe_schema str
    Schema of the resulting dataframe after transformations, in Spark StructType JSON format (from df.schema.json()). Required if transformationSql is specified. Example: {"type":"struct","fields":[{"name":"colA","type":"integer","nullable":true,"metadata":{}},{"name":"colC","type":"integer","nullable":true,"metadata":{}}]}
    entity_columns Sequence[str]
    Deprecated: Use Feature.entity instead. Kept for backwards compatibility. The entity columns of the Delta table
    filter_condition str
    Single WHERE clause to filter delta table before applying transformations. Will be row-wise evaluated, so should only include conditionals and projections
    timeseries_column str
    Deprecated: Use Feature.timeseries_column instead. Kept for backwards compatibility. The timeseries column of the Delta table
    transformation_sql str
    A single SQL SELECT expression applied after filter_condition. Should contains all the columns needed (eg. "SELECT , colA + colB AS colC FROM x.y.z WHERE colA > 0" would have transformationSql ", colA + colB AS colC") If transformationSql is not provided, all columns of the delta table are present in the DataSource dataframe
    fullName String
    The full three-part (catalog, schema, table) name of the Delta table
    dataframeSchema String
    Schema of the resulting dataframe after transformations, in Spark StructType JSON format (from df.schema.json()). Required if transformationSql is specified. Example: {"type":"struct","fields":[{"name":"colA","type":"integer","nullable":true,"metadata":{}},{"name":"colC","type":"integer","nullable":true,"metadata":{}}]}
    entityColumns List<String>
    Deprecated: Use Feature.entity instead. Kept for backwards compatibility. The entity columns of the Delta table
    filterCondition String
    Single WHERE clause to filter delta table before applying transformations. Will be row-wise evaluated, so should only include conditionals and projections
    timeseriesColumn String
    Deprecated: Use Feature.timeseries_column instead. Kept for backwards compatibility. The timeseries column of the Delta table
    transformationSql String
    A single SQL SELECT expression applied after filter_condition. Should contains all the columns needed (eg. "SELECT , colA + colB AS colC FROM x.y.z WHERE colA > 0" would have transformationSql ", colA + colB AS colC") If transformationSql is not provided, all columns of the delta table are present in the DataSource dataframe

    FeatureEngineeringKafkaConfigIngestionConfigIngestionDestination, FeatureEngineeringKafkaConfigIngestionConfigIngestionDestinationArgs

    FeatureEngineeringKafkaConfigKeySchema, FeatureEngineeringKafkaConfigKeySchemaArgs

    JsonSchema string
    Schema of the JSON object in standard IETF JSON schema format (https://json-schema.org/)
    JsonSchema string
    Schema of the JSON object in standard IETF JSON schema format (https://json-schema.org/)
    json_schema string
    Schema of the JSON object in standard IETF JSON schema format (https://json-schema.org/)
    jsonSchema String
    Schema of the JSON object in standard IETF JSON schema format (https://json-schema.org/)
    jsonSchema string
    Schema of the JSON object in standard IETF JSON schema format (https://json-schema.org/)
    json_schema str
    Schema of the JSON object in standard IETF JSON schema format (https://json-schema.org/)
    jsonSchema String
    Schema of the JSON object in standard IETF JSON schema format (https://json-schema.org/)

    FeatureEngineeringKafkaConfigProviderConfig, FeatureEngineeringKafkaConfigProviderConfigArgs

    WorkspaceId string
    Workspace ID which the resource belongs to. This workspace must be part of the account which the provider is configured with.
    WorkspaceId string
    Workspace ID which the resource belongs to. This workspace must be part of the account which the provider is configured with.
    workspace_id string
    Workspace ID which the resource belongs to. This workspace must be part of the account which the provider is configured with.
    workspaceId String
    Workspace ID which the resource belongs to. This workspace must be part of the account which the provider is configured with.
    workspaceId string
    Workspace ID which the resource belongs to. This workspace must be part of the account which the provider is configured with.
    workspace_id str
    Workspace ID which the resource belongs to. This workspace must be part of the account which the provider is configured with.
    workspaceId String
    Workspace ID which the resource belongs to. This workspace must be part of the account which the provider is configured with.

    FeatureEngineeringKafkaConfigSubscriptionMode, FeatureEngineeringKafkaConfigSubscriptionModeArgs

    Assign string
    A JSON string that contains the specific topic-partitions to consume from. For example, for '{"topicA":[0,1],"topicB":[2,4]}', topicA's 0'th and 1st partitions will be consumed from
    Subscribe string
    A comma-separated list of Kafka topics to read from. For example, 'topicA,topicB,topicC'
    SubscribePattern string
    A regular expression matching topics to subscribe to. For example, 'topic.*' will subscribe to all topics starting with 'topic'
    Assign string
    A JSON string that contains the specific topic-partitions to consume from. For example, for '{"topicA":[0,1],"topicB":[2,4]}', topicA's 0'th and 1st partitions will be consumed from
    Subscribe string
    A comma-separated list of Kafka topics to read from. For example, 'topicA,topicB,topicC'
    SubscribePattern string
    A regular expression matching topics to subscribe to. For example, 'topic.*' will subscribe to all topics starting with 'topic'
    assign string
    A JSON string that contains the specific topic-partitions to consume from. For example, for '{"topicA":[0,1],"topicB":[2,4]}', topicA's 0'th and 1st partitions will be consumed from
    subscribe string
    A comma-separated list of Kafka topics to read from. For example, 'topicA,topicB,topicC'
    subscribe_pattern string
    A regular expression matching topics to subscribe to. For example, 'topic.*' will subscribe to all topics starting with 'topic'
    assign String
    A JSON string that contains the specific topic-partitions to consume from. For example, for '{"topicA":[0,1],"topicB":[2,4]}', topicA's 0'th and 1st partitions will be consumed from
    subscribe String
    A comma-separated list of Kafka topics to read from. For example, 'topicA,topicB,topicC'
    subscribePattern String
    A regular expression matching topics to subscribe to. For example, 'topic.*' will subscribe to all topics starting with 'topic'
    assign string
    A JSON string that contains the specific topic-partitions to consume from. For example, for '{"topicA":[0,1],"topicB":[2,4]}', topicA's 0'th and 1st partitions will be consumed from
    subscribe string
    A comma-separated list of Kafka topics to read from. For example, 'topicA,topicB,topicC'
    subscribePattern string
    A regular expression matching topics to subscribe to. For example, 'topic.*' will subscribe to all topics starting with 'topic'
    assign str
    A JSON string that contains the specific topic-partitions to consume from. For example, for '{"topicA":[0,1],"topicB":[2,4]}', topicA's 0'th and 1st partitions will be consumed from
    subscribe str
    A comma-separated list of Kafka topics to read from. For example, 'topicA,topicB,topicC'
    subscribe_pattern str
    A regular expression matching topics to subscribe to. For example, 'topic.*' will subscribe to all topics starting with 'topic'
    assign String
    A JSON string that contains the specific topic-partitions to consume from. For example, for '{"topicA":[0,1],"topicB":[2,4]}', topicA's 0'th and 1st partitions will be consumed from
    subscribe String
    A comma-separated list of Kafka topics to read from. For example, 'topicA,topicB,topicC'
    subscribePattern String
    A regular expression matching topics to subscribe to. For example, 'topic.*' will subscribe to all topics starting with 'topic'

    FeatureEngineeringKafkaConfigValueSchema, FeatureEngineeringKafkaConfigValueSchemaArgs

    JsonSchema string
    Schema of the JSON object in standard IETF JSON schema format (https://json-schema.org/)
    JsonSchema string
    Schema of the JSON object in standard IETF JSON schema format (https://json-schema.org/)
    json_schema string
    Schema of the JSON object in standard IETF JSON schema format (https://json-schema.org/)
    jsonSchema String
    Schema of the JSON object in standard IETF JSON schema format (https://json-schema.org/)
    jsonSchema string
    Schema of the JSON object in standard IETF JSON schema format (https://json-schema.org/)
    json_schema str
    Schema of the JSON object in standard IETF JSON schema format (https://json-schema.org/)
    jsonSchema String
    Schema of the JSON object in standard IETF JSON schema format (https://json-schema.org/)

    Package Details

    Repository
    databricks pulumi/pulumi-databricks
    License
    Apache-2.0
    Notes
    This Pulumi package is based on the databricks Terraform Provider.
    databricks logo
    Viewing docs for Databricks v1.96.0
    published on Thursday, Jun 18, 2026 by Pulumi
    Schema (JSON)
    pulumi/pulumi-databricks

      Try Pulumi Cloud free.
      Your team will thank you.

      Start free trial