Should Maho have a generic async queue layer?

[hirale]

Looking at PR #915's AI module, I noticed the async task queue (Mage::helper('ai')->submitTask() + cron processing). It got me thinking — that pattern could be useful beyond AI tasks.

There are plenty of other use cases that could benefit from the same async processing:

• Emails, webhooks
• Imports/exports, external API syncing
• Third-party tracking events
• Image processing, search indexing

I've built something similar before for OpenMage (redis-queue), mostly for async tracking.

A generic Maho_Queue could start simple — database-backed by default, cron as the worker — and scale to Redis or other backends later.

The question: Is there enough demand to extract this as a reusable core layer, or is it better to keep async patterns module-specific for now?

[fballiano]

Hi @hirale! I had your same thought checking #915 :-)

I've always wondered what was the real difference between having a queue or just process stuff via a cron job, is it really that different?

[hirale]

Great question — I think cron and queue solve different layers of the problem.

Maho's cron already has locking, and that's good. The current code locks a cron_schedule row atomically by flipping its status from pending to running, so two cron runners won't execute the same schedule at the same time.

But that lock is schedule-level, not work-item-level.

Take an erp_export_orders job that runs every 5 minutes and scans unsynced orders. Cron can lock the schedule row for that run — but it knows nothing about the 100 orders inside it. It can't tell which order is processing, which one failed, how many attempts it's had, when it should be retried, whether it's locked until a future time, or whether it should be dead-lettered.

And there's no per-item retry at the cron level either. The cron_schedule table has schedule_id, job_code, status, timestamps, and messages — no attempts, max_attempts, next_attempt_at, or locked_until. When _processJob() catches an exception, it marks the row as error and moves on. The only "retry" is that the next cron cycle generates a brand-new cron_schedule row for the same job_code — but that's a new schedule, not a retry of the failed work items inside the previous run.

This pattern already shows up in several core modules:

• Email Automation tracks each customer's progress through a multi-step campaign. Every "send customer X the step-Y email" is an independent work item — if SMTP fails for one customer, you need to know which customer, which step, and how many times it's been attempted. The module already has its own per-customer state machine and cron processing to handle this.
• FeedManager exports feeds to Google Shopping, Meta, Pinterest, etc. via SFTP/FTP. If the Google Shopping feed uploads fine but the Meta SFTP times out, you want to retry just that one delivery — not regenerate and re-upload every feed.
• core_email_queue is almost a counter-example — it has processed_at but no retry count, no max attempts, no backoff. A failed email just sits there waiting for the next sweep, with no visibility into how many times it's already failed or whether it should be given up on.

PR #915 follows the same pattern. The async part isn't just a cron job — it has maho_ai_task, submitTask(), pending → processing → complete / failed / cancelled, max_retries, callbacks, stuck-processing recovery after 120 seconds, cleanup, and an admin task grid.

Each of these modules is solving (or ignoring) the same problem independently. A Maho_Queue would standardize the durable work-item contract so they don't each have to reinvent their own queue table and retry/locking logic.

I wouldn't frame this as "queue instead of cron" — the simple version can still be database-backed and processed by cron. The difference is what sits between the cron runner and the actual work.

TL;DR:
Cron → "run this maintenance job every N minutes."
Queue → "process many independent work items, each with its own payload, status, retry policy, lock, failure history, and recovery behavior."

[fballiano]

mmmm interesting, I'll try to think about this more in the next few weeks :-)