Mark DBFS tasks as legacy in favor of Unity Catalog Volumes

[fdelbrayelle]

Summary

Databricks now documents DBFS root and DBFS mounts as deprecated and not recommended, steering users toward Unity Catalog volumes, external locations, or workspace files instead. This issue marks plugin-databricks's existing dbfs.Upload/dbfs.Download tasks as legacy in their documentation and @Schema descriptions, pointing users to the new Unity Catalog Volumes tasks (#250) — without removing or breaking the existing tasks.

Motivation

New Databricks workspaces are already provisioned without DBFS root/mount access by default, and Databricks' own "Best practices for DBFS and Unity Catalog" guidance is unambiguous about steering customers away from DBFS. Users building new flows against plugin-databricks today have no signal, inside Kestra itself, that they should reach for Unity Catalog Volumes instead of dbfs.Upload/dbfs.Download — they'd only discover this by separately reading Databricks' docs. A clear in-product legacy notice (docs + @Schema wording) closes that gap cheaply, while intentionally stopping short of a hard Java @Deprecated or removal, since DBFS and UC Volumes are documented to coexist and many existing flows depend on the current tasks working unchanged.

Context

Part of the Databricks platform coverage EPIC: https://github.com/kestra-io/kestra-ee/issues/9384

Depends on #250 ("Add Unity Catalog Support Plugin") — the unitycatalog.volume Upload/Download tasks introduced there are what this issue's documentation points users toward. This issue should not be started until #250's Volumes tasks exist (or at least are far enough along that the doc cross-links resolve).

Databricks' own guidance:

• https://docs.databricks.com/aws/en/dbfs/unity-catalog — "Best practices for DBFS and Unity Catalog" (explicit deprecation language)
• https://docs.databricks.com/aws/en/dbfs — "What is DBFS?"
• https://docs.databricks.com/aws/en/dbfs/dbfs-root — recommendations for DBFS root

API Reference

• Official docs: https://docs.databricks.com/aws/en/dbfs/unity-catalog
• Authentication: unchanged — this issue makes no API or auth changes, only documentation/schema wording
• Base URL pattern: unchanged (/api/2.0/dbfs/... for the existing tasks; no code path changes)
• SDK / client library: none — no new dependency

Gradle Dependencies

None. This issue only touches @Schema descriptions and the plugin how-to doc — no new dependency, no behavior change, no version bump beyond what #250 already introduces.

Plugin Structure

• Repository: plugin-databricks (existing)
• Namespace: io.kestra.plugin.databricks.dbfs (existing — no new sub-package)
• Sub-plugins: none (documentation-only change to the existing dbfs sub-package)
• Categories: DATA (unchanged)

Suggested Tasks

1. Update dbfs.Upload and dbfs.Download's class-level @Schema(title = ...) / description to note that Databricks considers DBFS legacy and to recommend the new unitycatalog.volume.Upload/Download tasks (from Add Unity Catalog Support Plugin #250) for new flows — wording only, do not add a Java @Deprecated annotation (would surface as a compiler/IDE warning on every existing flow using these tasks, which is not the intent here)
2. Update io.kestra.plugin.databricks.md's ## Tasks section for the dbfs sub-group with a short callout: DBFS is legacy per Databricks' own guidance, link to the Unity Catalog Volumes tasks as the recommended path for new flows
3. Add a one-line note to README.md if it separately documents the dbfs tasks
4. No test changes required — behavior is unchanged; existing dbfs tests must continue to pass as-is

YAML Examples

Note: this issue does not add a new task or trigger type, so the two examples below show the migration path (existing DBFS flow vs. its Unity Catalog Volumes equivalent) rather than a novel capability. No trigger example applies — neither dbfs nor this issue touches a trigger.

Example 1 — Existing DBFS upload flow (still supported, now documented as legacy)

id: dbfs_upload_legacy
namespace: company.team

tasks:
  - id: upload_report
    type: io.kestra.plugin.databricks.dbfs.Upload
    host: "{{ secret('DATABRICKS_HOST') }}"
    token: "{{ secret('DATABRICKS_TOKEN') }}"
    from: "{{ outputs.generate_report.uri }}"
    to: "dbfs:/reports/daily_report.csv"

Example 2 — Recommended Unity Catalog Volumes equivalent

id: uc_volume_upload_recommended
namespace: company.team

tasks:
  - id: upload_report
    type: io.kestra.plugin.databricks.unitycatalog.volume.Upload
    workspaceHost: "{{ secret('DATABRICKS_HOST') }}"
    token: "{{ secret('DATABRICKS_TOKEN') }}"
    from: "{{ outputs.generate_report.uri }}"
    volumePath: "/Volumes/main/reports/daily/daily_report.csv"

Acceptance Criteria

Functional

• dbfs.Upload/dbfs.Download behavior is completely unchanged — no @Deprecated annotation added
• @Schema descriptions updated with legacy wording and a link to the Unity Catalog Volumes replacement tasks
• Existing dbfs unit/integration tests continue to pass unmodified

Kestra Plugin Coding Standards

• No new properties introduced — this is a documentation-only change
• @Schema descriptions remain accurate and reflect the current (unchanged) behavior alongside the legacy notice

Documentation & Structure

• io.kestra.plugin.databricks.md ## Tasks section for dbfs updated with the legacy callout and cross-link to Unity Catalog Volumes
• README.md updated if it separately documents dbfs usage

---

View as Artifact

[fdelbrayelle]

Depends on #250 (Unity Catalog Volumes Upload/Download tasks) — start after that lands.