Add deprecation warnings for Expr passed to confirmed literal-only function arguments#1605
Merged
Merged
Conversation
- Introduced shared `_warn_if_expr_for_literal_arg` in `functions/__init__.py` - Added `DeprecationWarning` for the following methods when `Expr` is passed as argument: - `encode(..., encoding=Expr)` - `decode(..., encoding=Expr)` - `digest(..., method=Expr)` - `arrow_cast(..., data_type=Expr)` - `arrow_try_cast(..., data_type=Expr)` - `arrow_metadata(..., key=Expr)` test: update tests to check for warnings - Implemented tests in `test_functions.py` to ensure: - Warning is raised for `Expr` form - No warning for native literal form
nuno-faria
approved these changes
Jun 19, 2026
nuno-faria
left a comment
nuno-faria
left a comment
Contributor
There was a problem hiding this comment.
Thanks @kosiew. In addition to the comments below, it's better to update the arrow_cast calls at user-guide/common-operations/functions.md which are still using expressions.
Comment on lines
+67
to
+82
| if isinstance(value, Expr): | ||
| _warn_expr_for_literal_arg(function_name, arg_name) | ||
|
|
||
|
|
Contributor
There was a problem hiding this comment.
Wouldn't it be better to combine these two into a single function?
Comment on lines
1603
to
1619
| def test_binary_string_functions(df): | ||
| df = df.select( | ||
| f.encode(column("a").cast(pa.string()), literal("base64").cast(pa.string())), | ||
| f.decode( | ||
| f.encode( | ||
| column("a").cast(pa.string()), literal("base64").cast(pa.string()) | ||
| ), | ||
| literal("base64").cast(pa.string()), | ||
| ), | ||
| ) | ||
| result = df.collect() | ||
| assert len(result) == 1 | ||
| result = result[0] | ||
| assert result.column(0) == pa.array(["SGVsbG8", "V29ybGQ", "IQ"]) | ||
| assert pa.array(result.column(1)).cast(pa.string()) == pa.array( | ||
| ["Hello", "World", "!"] | ||
| ) |
Contributor
There was a problem hiding this comment.
The calls to encode and decode need to be updated to not raise warnings.
| assert result.column(0) == expected_result | ||
|
|
||
|
|
||
| def test_hash_functions(df): |
Contributor
There was a problem hiding this comment.
The calls here also need to be updated.
Contributor
Author
There was a problem hiding this comment.
Changed:
- python/datafusion/functions/init.py
- collapsed warning helpers into single
_warn_if_expr_for_literal_arg - updated temporal _date_part / _date_trunc callers
- collapsed warning helpers into single
- python/tests/test_functions.py
- digest behavior test now uses native method strings
- encode / decode behavior test now uses native "base64"
Changed docs/source/user-guide/common-operations/functions.md:
- arrow_cast(..., string_literal("Float64")) → arrow_cast(...,
"Float64") - arrow_cast(..., str_lit("Int32")) → arrow_cast(..., "Int32")
- removed unused string_literal, str_lit imports.
- Collapsed warning helpers into a single function `_warn_if_expr_for_literal_arg` in `python/datafusion/functions/__init__.py`. - Updated callers of temporal functions `_date_part` and `_date_trunc` in `python/datafusion/functions/__init__.py`. - Modified the digest behavior test in `python/tests/test_functions.py` to use native method strings. - Updated the encode/decode behavior test to use native "base64" in `python/tests/test_functions.py`.
- Replaced `string_literal("Float64")` with `"Float64"` in examples.
- Replaced `str_lit("Int32")` with `"Int32"` in examples.
- Removed unused `string_literal` and `str_lit` imports.
Contributor
Author
|
Thanks @nuno-faria for your review |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
As part of the audit of
Expr | literalfunction arguments, this change targets a set of arguments that act as control/configuration parameters rather than row-varying data expressions. These arguments are intended to be provided as Python literals (for example, encoding names, digest methods, data type specifiers, and metadata keys).To prepare for future API cleanup while preserving compatibility, passing an
Exprto these arguments now emits aDeprecationWarningbut continues to work.What changes are included in this PR?
Added a reusable helper,
_warn_if_expr_for_literal_arg, that emits a deprecation warning when anExpris supplied to a confirmed literal-only argument.Added deprecation warnings for the following function arguments:
encode(..., encoding=...)decode(..., encoding=...)digest(..., method=...)arrow_cast(..., data_type=...)arrow_try_cast(..., data_type=...)arrow_metadata(..., key=...)Preserved existing behavior by continuing to coerce these values to expressions after warning.
Added tests covering both warning and non-warning cases.
Are these changes tested?
Yes.
Added tests in
python/tests/test_functions.pythat verify:Passing
literal(...)/Exprvalues to the audited literal-only arguments emits aDeprecationWarning.Passing native Python literal values does not emit a
DeprecationWarning.Coverage includes:
encodedecodedigestarrow_castarrow_try_castarrow_metadataAre there any user-facing changes?
Yes.
Users will now receive a
DeprecationWarningwhen passing anExprto the following literal-only arguments:encode(..., encoding=...)decode(..., encoding=...)digest(..., method=...)arrow_cast(..., data_type=...)arrow_try_cast(..., data_type=...)arrow_metadata(..., key=...)Existing behavior is otherwise unchanged, and Python literal inputs remain fully supported without warnings.
LLM-generated code disclosure
This PR includes code, comments generated with assistance from LLM. All LLM-generated content has been manually reviewed.