Fix: Graceful Startup Config Retries & ML Load Fallback by rholligan · Pull Request #4216 · springfall2008/batpred

rholligan · 2026-07-09T12:55:42Z

This PR resolves several startup initialization bugs when dealing with slower Home Assistant environments and unexpected ML model outputs.

Changes:

Auto-Config Retries: Fixed a bug where unmatched regex strings (re:sensor...) in apps.yaml lists/dicts were destructively overwritten with None. They now retain their original string and are cleanly retried.
Startup Entity Buffer: predbat.py now waits 10 minutes from startup before finalizing auto_config (hard-disabling unmatched entities), allowing slower HA environments time to load all integrations.
ML Load NaN Safety: Added safe_float to utils.py and applied it to the ML component to prevent InvalidJSONError crashes when the ML model occasionally outputs NaN or Inf.
Tests & CI: Added 100% test coverage for the new retry/fallback logic. Also squashed several long-standing cross-platform test bugs (Unicode emojis and CRLF line-endings) that caused unit_test.py to crash on Windows environments.

…tartup

…ecast fallback when model is invalid

…cuting >10 minutes after startup to prevent permanent dummy regex strings

…retaining original strings instead of replacing with None

…s fallback

…suite crashes

… to crlf line endings

Copilot

Pull request overview

This PR targets Predbat startup resilience by (1) improving auto_config handling of regex-based entity configuration during slow Home Assistant startups, and (2) hardening ML load forecasting publication/consumption against non-JSON-compliant floats, alongside several cross-platform test-output cleanups.

Changes:

Add safe_float() and apply it to ML load stats publishing to avoid crashes when values become NaN/Inf.
Rework regex resolution and startup retry timing for auto_config, plus add new tests intended to cover the retry/fallback paths.
Normalize many unit-test prints/comments to avoid Unicode/CRLF issues on Windows.

Reviewed changes

Copilot reviewed 52 out of 53 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
apps/predbat/utils.py	Adds `safe_float()` helper for sanitizing floats.
apps/predbat/userinterface.py	Changes regex resolution and `auto_config` retry bookkeeping.
apps/predbat/unit_test.py	Registers new test modules in the test runner.
apps/predbat/tests/test_temperature.py	Replaces Unicode status markers with ASCII text.
apps/predbat/tests/test_storage.py	Comment-only ASCII normalization.
apps/predbat/tests/test_solis.py	Comment-only ASCII normalization.
apps/predbat/tests/test_solcast.py	Comment-only ASCII normalization.
apps/predbat/tests/test_sigenergy.py	Comment/docstring ASCII normalization.
apps/predbat/tests/test_rate_replicate_missing_slots.py	Replaces Unicode status markers with ASCII text.
apps/predbat/tests/test_rate_min_forward_calc.py	Comment-only ASCII normalization.
apps/predbat/tests/test_rate_add_io_slots.py	Comment-only ASCII normalization.
apps/predbat/tests/test_open_meteo.py	Docstring/comment ASCII normalization.
apps/predbat/tests/test_ohme.py	Replaces Unicode status markers with ASCII text.
apps/predbat/tests/test_octopus_url.py	Replaces Unicode status markers with ASCII text.
apps/predbat/tests/test_octopus_read_response_retry.py	Replaces Unicode status markers with ASCII text.
apps/predbat/tests/test_octopus_misc.py	Replaces Unicode status markers with ASCII text.
apps/predbat/tests/test_octopus_download_rates.py	Replaces Unicode status markers with ASCII text.
apps/predbat/tests/test_octopus_day_night_rates.py	Docstring/comment ASCII normalization + status text changes.
apps/predbat/tests/test_ml_load_fallback.py	New tests for ML load fallback behaviour.
apps/predbat/tests/test_minute_data.py	Comment-only ASCII normalization.
apps/predbat/tests/test_minute_array.py	Comment-only ASCII normalization.
apps/predbat/tests/test_load_ml.py	Test output ASCII normalization (and some comment tweaks).
apps/predbat/tests/test_kraken.py	Comment-only ASCII normalization.
apps/predbat/tests/test_integer_config.py	Replaces Unicode status markers with ASCII text.
apps/predbat/tests/test_hainterface_websocket.py	Replaces Unicode status markers with ASCII text.
apps/predbat/tests/test_hainterface_state.py	Replaces Unicode status markers with ASCII text.
apps/predbat/tests/test_hainterface_service.py	Replaces Unicode status markers with ASCII text.
apps/predbat/tests/test_hainterface_lifecycle.py	Replaces Unicode status markers with ASCII text.
apps/predbat/tests/test_hainterface_api.py	Replaces Unicode status markers with ASCII text.
apps/predbat/tests/test_ge_cloud.py	Replaces Unicode status markers with ASCII text.
apps/predbat/tests/test_gateway.py	Docstring/comment ASCII normalization.
apps/predbat/tests/test_futurerate_auto.py	Docstring ASCII normalization.
apps/predbat/tests/test_find_charge_window.py	Docstring/comment ASCII normalization.
apps/predbat/tests/test_find_battery_size.py	Comment-only ASCII normalization.
apps/predbat/tests/test_fill_load_from_power.py	Replaces Unicode status markers with ASCII text.
apps/predbat/tests/test_fetch_pv_forecast.py	Replaces Unicode status markers with ASCII text.
apps/predbat/tests/test_fetch_config_options.py	Replaces Unicode status markers with ASCII text.
apps/predbat/tests/test_download.py	Fixes temp file writing to be bytes-mode (Windows-friendly).
apps/predbat/tests/test_db_manager.py	Replaces Unicode status markers with ASCII text.
apps/predbat/tests/test_component_base.py	Comment-only ASCII normalization.
apps/predbat/tests/test_compare.py	Comment-only ASCII normalization.
apps/predbat/tests/test_carbon.py	Replaces Unicode status markers with ASCII text.
apps/predbat/tests/test_battery_curve_keys.py	Replaces Unicode status markers with ASCII text.
apps/predbat/tests/test_balance_inverters.py	Replaces Unicode status markers with ASCII text.
apps/predbat/tests/test_auto_config.py	New tests for `auto_config` regex retry behaviour.
apps/predbat/tests/open_meteo_live.py	Comment-only ASCII normalization.
apps/predbat/predbat.py	Changes startup `auto_config` finalization and adds a 10-minute retry window.
apps/predbat/load_ml_component.py	Uses `safe_float()` for ML stats publishing to avoid JSON serialization crashes.
apps/predbat/fetch.py	Adds “inactive ML forecast” early return fallback to historical data.

Comments suppressed due to low confidence (1)

apps/predbat/userinterface.py:1099

auto_config(final=False) does nothing when an arg doesn’t match (it neither disables the arg nor records it in unmatched_args). Since predbat.py only retries auto_config when unmatched_args is non-empty, unmatched regex args won’t be retried during startup, which contradicts the intended “retry for 10 minutes then final-disable” behaviour.

            for item_value in arg_value:
                item_matched, item_value = self.resolve_arg_re(arg, item_value, state_keys)
                if not item_matched:
-                    self.log("Warn: Regular argument {} expression {} failed to match - disabling this item".format(arg, item_value))
-                    new_list.append(None)
-                else:
-                    new_list.append(item_value)
+                    self.log("Warn: Regular argument {} expression {} failed to match - will retry".format(arg, item_value))
+                new_list.append(item_value)


                item_value = arg_value[item_name]
                item_matched, item_value = self.resolve_arg_re(arg, item_value, state_keys)
                if not item_matched:
-                    self.log("Warn: Regular argument {} expression {} failed to match - disabling this item".format(arg, item_value))
-                    new_dict[item_name] = None
-                else:
-                    new_dict[item_name] = item_value
+                    self.log("Warn: Regular argument {} expression {} failed to match - will retry".format(arg, item_value))
+                new_dict[item_name] = item_value


 from datetime import datetime, timezone, timedelta
+import math
 from component_base import ComponentBase


+    # Test 1: Active status proceeds
+    fetch.prefix = "active_test"
+    try:
+        fetch.fetch_ml_load_forecast(now_utc)
+        print("FAIL: ML Load forecast should have raised AttributeError on active path")
+        failed = True
+    except AttributeError:
+        # It bypassed the early return and hit unmocked attributes, which means it works
+        print("PASS: ML Load forecast active status proceeds past early return")
+
+    # Test 2: Inactive status falls back
+    fetch.prefix = "inactive_test"
+    fetch.load_ml_forecast = {}
+    try:
+        fetch.fetch_ml_load_forecast(now_utc)
+        if fetch.load_ml_forecast:
+            print("FAIL: ML Load forecast should NOT be populated when inactive")
+            failed = True
+        else:
+            print("PASS: ML Load forecast ignored when inactive (fallback)")
+    except AttributeError:
+        print("FAIL: ML Load forecast did NOT fall back early and hit AttributeError")
+        failed = True


…ows tests

…an generation output from main

rholligan added 13 commits July 9, 2026 11:39

fix: delay auto_config finalization to allow HA entities to load on s…

9575534

…tartup

fix: resolve NaN InvalidJSONError in ML component and handle load for…

4b7fc3e

…ecast fallback when model is invalid

fix: correctly apply final=True logic to auto config on any loops exe…

599fa1a

…cuting >10 minutes after startup to prevent permanent dummy regex strings

fix: correctly retry regex config elements inside lists and dicts by …

cc323f9

…retaining original strings instead of replacing with None

test: add test coverage for auto_config retry logic and ML load statu…

1cd3eb9

…s fallback

test: fix mock environment for ml load fallback test

0fc0d81

test: fix mock environment for ml load fallback test

ca4cad3

refactor: simplify and clean up fixes for repo consistency

51dfd1f

test: replace unicode checkmarks with ascii text to fix windows test …

1e0c4bf

…suite crashes

test: replace more unicode checkmarks to fix windows suite crashes

602d4d2

test: replace unicode arrows with ascii text

28cdcd3

test: replace final straggler unicode emojis with ascii

6b5b0a7

test: fix test_download.py sha1 hash assertion failing on windows due…

a45c5f5

… to crlf line endings

springfall2008 requested a review from Copilot July 9, 2026 17:56

Copilot started reviewing on behalf of springfall2008 July 9, 2026 17:56 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

rholligan added 4 commits July 9, 2026 19:53

fix: address PR review feedback from copilot

138a267

test: replace literal escaped unicode strings with ascii equivalents

ceff435

test: fix unicode right arrow in ML load predictor logs crashing wind…

27ce793

…ows tests

test: update expected json for pre_saving1 debug case to match new pl…

d241e69

…an generation output from main

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix: Graceful Startup Config Retries & ML Load Fallback#4216

Fix: Graceful Startup Config Retries & ML Load Fallback#4216
rholligan wants to merge 17 commits into
springfall2008:mainfrom
rholligan:fix/auto-config-startup-retry

rholligan commented Jul 9, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

rholligan commented Jul 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants