Skip to content

CHANGELOG.md

Latest commit

 

History

History
1016 lines (762 loc) · 44.2 KB

CHANGELOG.md

File metadata and controls

1016 lines (762 loc) · 44.2 KB
Raw

Changelog

master

View commits

3.4.0

  • Backwards-incompatible. The django stats app writes to its own sqlite database by default (huey-stats.db in settings.BASE_DIR), instead of DATABASES['default']. To keep recording in your main Django database, specify the database explicitly:

    HUEY_STATS = {'database': 'postgresql://user:password@localhost/my_db'}

    HUEY_STATS['database'] takes a db-url or a peewee Database, or use HUEY_STATS['filename'] to set the sqlite path. See the Django docs section for building a peewee database from your existing DATABASES entry.

    The admin Events log is now rendered from the stats database through peewee, rather than being a ModelAdmin via ORM. It is now linked from the dashboard's Recent events heading.

Consumer:

  • Add --shutdown-timeout option to bound graceful shutdown time.
  • Add --graceful-signal option to allow making SIGTERM the graceful shutdown signal (e.g. -g TERM).

User-facing APIs:

  • Add ttl to lock_task() and put_if_empty() so locks expire on their own. Supported by memory and redis storages, others raise NotImplementedError. RedisHuey requires server hash-field TTL support (redis 7.4+ or valkey 9+), RedisExpireHuey works on any version.
  • Add timeout parameter to aget_result().

Small stuff:

  • Ensure one raising signal receiver does not prevent the remaining receivers from running.
  • Fix crontab ranges with a step (0-30/5) and wrap-around ranges (22-2, day_of_week='1-7'). Raise ValueError when a field matches no values.
  • Pass failed chord members to the callback as huey.Error and skipped (revoked, expired, cancelled) members as huey.SKIPPED instead of a raw exception or None.
  • Delete chord partial results explicitly on completion, so they no longer linger until expiry on RedisExpireStorage.
  • Write the error of a failed pipeline stage to every skipped downstream stage, so their Result handles raise instead of blocking.
  • Accept an explicit id in TaskWrapper.s(), matching schedule().

Storage:

  • Skip messages that fail to deserialize in Huey.pending() and Huey.scheduled(), logging the error, as read_schedule() already does.
  • Add clean_name option to RedisHuey (default to True). Specify False to stop stripping non-alphanumeric characters from the storage namespace. The default will change in a future release.
  • Reduce Redis round-trips for peek_data, pop_data and delete_data, and make RedisExpireStorage.incr() set the TTL atomically w/ the increment.
  • Skip the exclusive lock in SqliteStorage.dequeue() when the queue is empty, so idle consumers no longer block producers and the scheduler.
  • Batch the SqliteStorage.read_schedule() delete to avoid sqlite's parameter limit when many tasks come due at once. Same fix in SqlHuey.
  • Order SqliteStorage.read_schedule() and SqlHuey.read_schedule() by (timestamp, id), matching PostgresStorage.
  • Add queue to the sqlite task index (task_queue_priority_id) so queues sharing a file no longer scan each other's rows. The old task_priority_id index is dropped when the schema is initialized. I'll get rid of the temporary in-band drop in a few releases.
  • Add peek_many() to the storage API. Revocation checks now fetch the task and task-class keys in one round trip.
  • Return str keys from result_items() on every storage backend. Redis and File storage previously returned bytes.

Stats:

  • Better handling for timeout, locked, rate-limited and retrying in stats recorder.
  • Log dropped events in the stats recorder instead of failing silently.
  • Fix stats recorder max_events pruning to count rows per queue. With multiple queues sharing a stats db, each queue retained only a fraction of the configured cap.
  • Add inflight_hours option to stats recorder, replacing the hardcoded 6 hour in-flight cutoff.
  • Require the change_hueydashboard permission for the django admin dashboard's revoke, restore and flush controls. Previously any staff user who could view the dashboard could also flush the queue.
  • Add huey[stats] extra which installs ol' peewee.

View commits

3.3.4

  • Use DoubleField for the stats ts, duration and started columns. On PG & MySQL the old FloatField was a 32-bit-shitter which lost a lot of seconds of precision. Existing PG/MySQL tables will need to run the SQL below to migrate OR just drop the tables and allow them to be rebuilt next run. SQLite was not affected. Fixes #909.
-- Postgres
alter table "huey_event" alter column "ts" type double precision;
alter table "huey_event" alter column "duration" type double precision;
alter table "huey_inflight" alter column "started" type double precision;

-- MySQL
alter table `huey_event` modify `ts` double not null;
alter table `huey_event` modify `duration` double;
alter table `huey_inflight` modify `started` double not null;

View commits

3.3.3

  • Forward the Django database OPTIONS when setting up the stats recorder connection. Fixes / replaces #907.
  • Remove accidental sqlite3 access at module scope, #908.

View commits

3.3.2

  • Flush buffered stats in the worker shutdown hook. Fixes issue where recycled workers might drop some writes.
  • Consolidate the stats flush to one inflight-table update per task, applied in sorted order. Concurrent writers (one per process worker) could deadlock in Postgres, and the failed flush dropped its whole batch of events.
  • Store the FileLock fd per-thread. Under thread contention the shared fd was clobbered and released by the wrong thread, leaking a held flock and wedging every process using that storage path. FileHuey with the default thread workers deadlocked on first contention.

View commits

3.3.1

  • Ensure traceback is stored properly for error results.
  • Restart the stats recorder in forked children via os.register_at_fork. Previously the writer thread existed only in the process that called enable_stats(), so a consumer w/ worker_type='process' (or a preforking web server that loads the app before forking) buffered task events in its workers but never wrote them, and the dashboard showed tasks stuck on "enqueued".

View commits

3.3.0

  • Add retry_backoff parameter to task() and periodic_task(). The first retry waits retry_delay seconds and each subsequent delay is multiplied by retry_backoff, giving exponentially-growing waits between retries.
  • Fix create_tables=False, which crashed SqliteHuey at connect and was silently ignored by the peewee SqlHuey. SqlStorage also gains initialize_schema() so the create_huey_tables command supports it.
  • Accept float priorities in FileStorage by truncating to int. Previously they raised a TypeError.
  • Raise ResultTimeout from blocking Result.get() when the wait ends w/o an obtainable result, e.g. a dropped connection. Previously the internal EmptyData sentinel could be returned.
  • Preserve per-call retry_backoff when a task is rescheduled via Result.reschedule().
  • Clear revocations that arrive mid-execution w/ a delete, so the clear also works on RedisExpireHuey where destructive reads do not remove data.
  • Fix an off-by-one in the redis scheduled_items() that returned limit+1 items.
  • Reconnect stale connections in the SqlHuey counter methods, which run in the consumer via chords and rate limits.
  • Return the lock from TaskLock.__enter__(), so with huey.lock_task('x') as lock: binds the lock instead of None.
  • Defer the redis server version check to first use. Previously every storage init issued an INFO round-trip.
  • Remove the undocumented Kyoto Tycoon storage backend, its tests, and the ukt CI dependency.
  • Remove the djhuey backend_class alias for huey_class (deprecated since 2.0) and the Django <1.2 settings.DATABASE_NAME queue-name fallback.
  • Fire the chord callback when a pipelined member dies before its tail. A failed, revoked, or expired stage now contributes for the member, where previously the chord counter stayed short and the callback never ran.
  • Add CySqliteHuey, which drives the sqlite storage w/ cysqlite instead of the stdlib sqlite3 module and takes an open-ended pragmas dict in place of a fixed set of tuning parameters.

View commits

3.2.1

  • Add a Django admin dashboard for task statistics. Adding huey.contrib.djhuey.stats to INSTALLED_APPS starts the huey.contrib.stats recorder in every process (incl. the consumer) and adds a Huey section to the admin: a dashboard w/ the same live stats and controls as the flask-peewee panel, plus a filterable event log. Stats are stored via peewee in the default Django database and require no migrations.

View commits

3.2.0

  • Add store_intermediate_errors option (default true, preserving current behavior). When false, a task that fails with retries remaining no longer writes its exception to the result store or runs its on_error handler until the retries are exhausted, so a blocking Result.get() waits for the final outcome instead of raising on the first failed attempt.
  • Add create_tables option to the SQL storage backends (default true). Pass create_tables=False to skip the automatic create table if not exists at init, e.g. to manage huey's schema via Django migrations rather than have every web worker attempt DDL on import.
  • Add a create_huey_tables Django management command to create the tables when create_tables=False.
  • Fix on_error handlers accumulating one exception argument per failed attempt across retries. Handlers now receives only the current attempt's exception.
  • Add huey.contrib.stats, a task-statistics engine: enable_stats(huey, db) records task signals into two peewee tables (huey_event, huey_inflight) and exposes a HueyStats query API for throughput, per-task timing, error-rates, in-flight and recent-event views. Depends only on peewee, so it can back a custom dashboard or exporter. Enable it in the consumer to capture task execution.
  • Add a Flask-Peewee admin panel, huey.contrib.flask_admin.HueyPanel, registered w/ admin.register_panel('Huey', HueyPanel, huey). It renders the recorded stats as a dashboard card plus a standalone page with live queue depths, throughput, per-task stats, running tasks and recent events. Has controls to revoke/restore tasks and flush the queue, schedule, results or locks. Requires flask-peewee 4.0.1+.

View commits

3.1.1

  • Ensure we use a safe name for long postgres queue names. PG has a 63 byte limit on the channel name.
  • Ensure recycled worker threads no longer leak their LISTEN connections w/Postgres.

View commits

3.1.0

  • Add first-class Postgres support: PostgresHuey. Workers use LISTEN/NOTIFY when a task is enqueued, giving Redis-like dequeue latency without polling, and dequeues use select ... for update skip locked so any number of consumers can share one database (requires psycopg 3.2+).
  • The django.tasks backend is now also compatible with the django-tasks backport package, extending support to pre-6.0 Django.
  • Use an explicit fork multiprocessing context for process workers, rather than setting the global start-method from the consumer entry-points. Fixes -k process on MacOS 3.8+ / Linux 3.14+ when the consumer is started via the huey_consumer console-script or a programmatic create_consumer().run().

View commits

3.0.3

  • Add a Django 6.0 task backend - pretty much works the same way the normal Django integration works (manage.py run_huey), but using Django's canonical APIs and decorator. Docs here.

View commits

3.0.2

  • Redis blocking dequeue no longer swallows ConnectionError: the error propagates to the worker, which logs it and applies exponential backoff. Previously a downed redis server caused workers to busy-loop silently.
  • Chord callbacks now fire when a member task is revoked, expired or cancelled by a pre-execute hook. The skipped member contributes a None placeholder result. Previously the callback was silently lost.
  • Scheduler skips missed periodic checks after a stall (e.g. suspend/resume) instead of running them back-to-back, which enqueued duplicate periodic tasks for the current minute.
  • Fix inverted timeout clamp in wait_result() when using notify_result with redis < 6 (or an unknown server version): timeouts over one second were cut to 1s, and sub-second timeouts blocked indefinitely.
  • put_if_empty() is now atomic for the memory and file storage backends, restoring lock_task() mutual exclusion on those backends.
  • FileLock no longer unlinks an existing lock file at construction time, which broke mutual exclusion for any process already holding the lock.
  • Process-worker task timeouts use signal.setitimer(), so float / sub-second timeouts work. Previously a timeout less than 1 second was silently ignored (alarm(0) cancels the timer) and fractional seconds were truncated.
  • Consumer signal handlers only set flags. Logging and greenlet cleanup now happen in the main loop, avoiding re-entrant I/O from signal context.
  • A user-supplied task kwarg named task is no longer dropped during serialization. Context tasks (context=True) inject the task instance into a copy of the kwargs rather than mutating the task's data.
  • MemoryStorage.dequeue() and add_to_schedule() acquire the storage lock, like the other mutating methods.
  • normalize_time() treats delay=0 as "now" rather than ignoring it, so e.g. expires=0 means "expires immediately" instead of "never expires".
  • Redis enqueued_items(limit) returned limit + 1 items from the producer end of the queue. It now returns the next-limit items to be dequeued, matching the other storage backends.
  • Redis-dependent tests are skipped when no local redis server is reachable, instead of failing at import time.

View commits

3.0.1

  • Fix bug in redis version parsing when using Elasticache or any other that sends major/minor. redis-py incorrectly parses these as floats because there's only a single decimal, so the version check was breaking.
  • Rename max task option --max-tasks (previously was --max_tasks).

View commits

3.0.0

  • Add chord() (map -> reduce) and group() (map) primitives.
  • Add timeout (using SIGALRM for process and gevent.Timeout for greenlet) to control task running time. For threads, unfortunately, there's no good mechanism so instead APIs for cooperatively checking timeout are provided on the Task instance.
  • Add simple fixed-window rate_limit() for tasks.
  • Add Result.is_ready() method for checking result readiness.
  • New option for low-latency result fetching, available for RedisHuey. To use, pass notify_result=True when initializing your Huey instance.
  • Add new incr(key, amount=1) to storage API for atomic increment primitive. This is used by chord().
  • Add new wait_result() method to storage APIs for efficiently waiting for a result to become ready. The default implementation uses the exponential backoff from the previous implementation of a blocking Result.get() - so no changes are needed. However if you have a custom storage implementation, this provides a mechanism for pub/sub or other notification of result readiness.
  • Remove Python 2.x compatibility.

View commits

2.6.0

  • "Modernize" build system to use pyproject.toml and github actions.

View commits

2.5.5

  • pypa/pypi is a joke.

View commits

2.5.4

  • Minor bug fixes
  • Fix multiprocessing start method for python 3.14+.

View commits

2.5.3

This release adds the oft-requested SIGNAL_ENQUEUED. This signal, of necessity, runs in the calling process and not in the consumer, since tasks are enqueued by the application typically. The exception is tasks that are enqueued for retry by the consumer or tasks (including periodic tasks) enqueued by the scheduler.

  • Add support for a new SIGNAL_ENQUEUED.
  • Use FOR UPDATE SKIP LOCKED when supported by the database in the sql_huey storage engine.

View commits

2.5.2

  • Prevent bad task serialization in schedule from causing a batch of tasks to be lost, see #815..
  • Ensure we catch ResultTimeout which may occur when used with Sentinel, #813.
  • Remove junk SQS implementation I was testing out.

2.5.1

  • More makework thanks to the ass-clowns running Python. Fix issue with deprecation of datetime.utcnow() in 3.12.
  • Add API for customizing the TaskWrapper implementation, suitably named get_task_wrapper_class().
  • Make the revoke_all(), restore_all() and is_revoked() more robust for various input types.
  • Fix bug that could occur in the event of a SIGHUP followed by a SIGINT when using thread workers.
  • Added new experimental contrib module for SQS queue and S3 result storage.

2.5.0

  • Check to ensure the gevent monkeypatch was applied when running the consumer with greenlet workers, log warning if it is not.
  • Explicitly clear the revoked flag on task instances after execution (#713). This will help reduce junk keys left in the storage if you attempt to revoke a task while it is executing.
  • Add support for delay=, eta= in Huey's .s() and .then() - this adds support for delaying or scheduling pipelines.
  • Add support for rescheduling callback pipelines when rescheduling a task. This is enabled by default (preserve_pipeline=True).
  • Add an on_commit_task() decorator for Django extension that will enqueue the task after any database changes have been committed. This eliminates a common race condition where a task is enqueued and executed before the corresponding database changes have been committed.
  • Allow overriding the delay and eta when raising a RetryTask exception. This provides finer-grained control over when a task should be retried.
  • Add a very simple ResultGroup.as_completed() helper to provide a way to deal with multiple results as they become available. Refs #746.
  • Add an asyncio helper for resolving task results asynchronously. Asyncio users can use await aget_result(result) or await aget_result_group(rg) to fetch a task result in non-blocking fashion.
  • Fix bug in SIGINT and SIGTERM behavior for gevent users.
  • Include lock name when a task fails due to TaskLocked exception (#757).

View commits

2.4.5

Improves propagation of errors in task results and includes fix for newer versions of pip.

View commits

2.4.4

  • Add is_locked(lock_name) to test whether lock is held.
  • Allow raising CancelExecution within a Task, and override retries.
  • Add a very basic redis-backed lock that can be acquired more than once (to provide a rudimentary semaphore).
  • Add a periodic_task() wrapper for MiniHuey class.

View commits

2.4.3

  • Fix compatibility with redis-py 4.0.0+.

View commits

2.4.2

  • Fix implementation of schedule-pop Lua script so it works with Redis cluster.
  • Ensure Django connections are closed before and after (previously they were only closed after) task execution with db_task() and db_periodic_task().
  • Allow additional lock-names to be specified when flushing locks.

View commits

2.4.1

  • Attempt to reconnect to database if connection becomes unusable (e.g. due to a server restart). See: huey.contrib.sql_huey.SqlHuey.
  • Do not use a soft file-lock for FileStorage - use fcntl.flock() instead.

View commits

2.4.0

View commits

2.3.2

  • Add hook (Huey.build_error_result) for customizing the error result metadata.
  • Avoid crashing if another module already modified/set the multiprocessing start method.

View commits

2.3.1

  • Add SIGNAL_INTERRUPTED to signal when a task is interrupted when a consumer exits abruptly.
  • Use the Huey.create_consumer() API within the Django management command, to allow Django users to customize the creation of the Consumer instance.

View commits

2.3.0

  • Use monotonic clock for timing operations within the consumer.
  • Ensure internal state is cleaned up on file-lock when released.
  • Support passing around TaskException as a pickled value.
  • Set the multiprocessing mode to "fork" on MacOS and Python 3.8 or newer.
  • Added option to enforce FIFO behavior when using Sqlite as storage.
  • Added the on_shutdown handler to djhuey namespace.
  • Ensure exception is set on AsyncResult in mini-huey.

View commits

2.2.0

  • Fix task repr (refs #460).
  • Adds task-id into metadata for task exceptions (refs #461).
  • Ensure database connection is not closed when using the call_local method of Django helper extension db_periodic_task().
  • Allow pickle protocol to be explicitly configured in serializer parameters.
  • Adds FileHuey and full FileStorage implementation.
  • Add shutdown() hook, which will be run in the context of the worker threads/processes during shutdown. This hook can be used to clean-up shared or global resources, for example.
  • Allow pipelines to be chained together. Additionally, support chaining task instances.

View commits

2.1.3

  • Fix semantics of SIGNAL_COMPLETE so that it is not sent until the result is ready.
  • Use classes for the specific Huey implementations (e.g. RedisHuey) so that it is easier to subclass / extend. Previously we just used a partial application of the constructor, which could be confusing.
  • Fix shutdown logic in consumer when using multiprocess worker model. Previously the consumer would perform a "graceful" shutdown, even when an immediate shutdown was requested (SIGTERM). Also cleans up the signal-handling code and ensures that interrupted tasks log a warning properly to indicate they were interrupted.

View commits

2.1.2

  • Allow AsyncResult object used in MiniHuey to support the __call__() method to block and resolve the task result.
  • When running the django run_huey management command, the huey loggers will not be configured if another logging handler is already registered to the huey namespace.
  • Added experimental contrib storage engine using kyoto tycoon <http://fallabs.com/kyototycoon>_ which supports task priority and the option to do automatic result expiration. Requires the ukt <https://github.com/coleifer/ukt>_ python package and a custom kyototycoon lua script.
  • Allow the Sqlite storage engine busy timeout to be configured when instantiating SqliteHuey.

View commits

2.1.1

  • Ensure that task()-decorated functions retain their docstrings.
  • Fix logger setup so that the consumer log configuration is only applied to the huey namespace, rather than the root logger.
  • Expose result, signal and disconnect_signal in the Django huey extension.
  • Add SignedSerializer, which signs and validates task messages.
  • Refactor the SqliteStorage so that it can be more easily extended to support other databases.

View commits

2.1.0

  • Added new contrib module sql_huey, which uses peewee <https://github.com/coleifer/peewee>_ to provide storage layer using any of the supported databases (sqlite, mysql or postgresql).
  • Added RedisExpireHuey, which modifies the usual Redis result storage logic to use an expire time for task result values. A consequence of this is that this storage implementation must keep all result keys at the top-level Redis keyspace. There are some small changes to the storage APIs as well, but will only possibly affect maintainers of alternative storage layers.
  • Also added a PriorityRedisExpireHuey which combines the priority-queue support from PriorityRedisHuey with the result-store expiration mechanism of RedisExpireHuey.
  • Fix gzip compatibility issue when using Python 2.x.
  • Add option to Huey to use zlib as the compression method instead of gzip.
  • Added FileStorageMethods storage mixin, which uses the filesystem for task result-store APIs (put, peek, pop).
  • The storage-specific Huey implementations (e.g. RedisHuey) are no longer subclasses, but instead are partial applications of the Huey constructor.

View commits

2.0.1

  • Small fixes, fixed typo in Exception class being caught by scheduler.

View commits

2.0.0

This section describes the changes in the 2.0.0 release. A detailed list of changes can be found here: https://huey.readthedocs.io/en/latest/changes.html

Overview of changes:

  • always_eager mode has been renamed to immediate mode. Unlike previous versions, immediate mode involves the same code paths used by the consumer process. This makes it easier to test features like task revocation and task scheduling without needing to run a dedicated consumer process. Immediate mode uses an in-memory storage layer by default, but can be configured to use "live" storage like Redis or Sqlite.
  • The events stream API has been removed in favor of simpler callback-driven signals APIs. These callbacks are executed synchronously within the huey consumer process.
  • A new serialization format is used in 2.0.0, however consumers running 2.0 will continue to be able to read and deserialize messages enqueued by Huey version 1.11.0 for backwards compatibility.
  • Support for task priorities.
  • New Serializer abstraction allows users to customize the serialization format used when reading and writing tasks.
  • Huey consumer and scheduler can be more easily run within the application process, if you prefer not to run a separate consumer process.
  • Tasks can now specify an on_error handler, in addition to the previously-supported on_complete handler.
  • Task pipelines return a special ResultGroup object which simplifies reading the results of a sequence of task executions.
  • SqliteHuey has been promoted out of contrib, onto an equal footing with RedisHuey. To simplify deployment, the dependency on peewee was removed and the Sqlite storage engine uses the Python sqlite3 driver directly.

View commits

1.11.0

Backwards-incompatible changes

Previously, it was possible for certain tasks to be silently ignored if a task with that name already existed in the registry. To fix this, I have made two changes:

  1. The task-name, when serialized, now consists of the task module and the name of the decorated function. So, "queue_task_foo" becomes "myapp.tasks.foo".
  2. An exception will be raised when attempting to register a task function with the same module + name.

Together, these changes are intended to fix problems described in #386.

Because these changes will impact the serialization (and deserialization) of messages, it is important that you consume all tasks (including scheduled tasks) before upgrading.

Always-eager mode changes

In order to provide a more consistent API, tasks enqueued using always_eager mode will now return a dummy TaskResultWrapper implementation that wraps the return value of the task. This change is designed to provide the same API for reading task result values, regardless of whether you are using always-eager mode or not.

Previously, tasks executed with always_eager would return the Python value directly from the task. When using Huey with the consumer, though, task results are not available immediately, so a special wrapper TaskResultWrapper is returned, which provides helper methods for retrieving the return value of the task. Going forward, always_eager tasks will return EagerTaskResultWrapper, which implements the same get() API that is typically used to retrieve task return values.

View commits

v1.10.5

  • Compatibility with redis-py 3.0, updated requirements / dependencies.
  • Add pre-/post- hooks into the djhuey namespace.

View commits

v1.10.4

  • Log time taken to execute tasks at default log level.
  • Fix missing import in SQLite storage backend.
  • Small refactoring in Redis storage backend to make it easier to override the driver / client implementation.
  • Fix failing tests for simpledb storage backend.

View commits

v1.10.3

  • Fixed regression where in always eager mode exceptions within tasks were being swallowed instead of raised.
  • Added an API for registering hooks to run when each worker process starts-up. This simplifies creating global/process-wide shared resources, such as a connection pool or database client. Documentation.

View commits

v1.10.2

  • More granular "extras" installation options.

View commits

v1.10.1

  • Remove call to SimpleDB Client.connect(), as the simpledb APIs have changed and no longer use this method.
  • Ensure that pre- and post-execute hooks are run when using Huey in "always_eager" mode.
  • Gracefully stop Huey consumer when SIGINT is received.
  • Improved continuous integration, now testing on Python 3.7 as well.

View commits

v1.10.0

  • Ensure that the default SIGINT handler is registered. This fixes an edge-case that arises when the consumer is run without job control, which causes interrupt signals to be ignored.
  • Restarts (SIGHUP) are now graceful by default.

View commits

v1.9.1

  • Ensure the scheduler loop does not drift (fixes #304).
  • Add TaskResultWrapper.reset() to enable resetting the results of tasks that failed and are subsequently being retried.
  • Allow task-decorated functions to be also decorated as periodic tasks.

View commits

v1.9.0

View commits

ROLLBACK of 1.8.0 Django Changes

Due to problems with the django patch that added support for multiple huey instances, I've decided to rollback those changes.

Django integration in Huey 1.9.0 will work the same as it had previously in 1.7.x and earlier.

Apologies, I should have reviewed the patch more thoroughly and insisted on better test coverage.

v1.8.0

View commits

Backwards-incompatible change to Django integration

NOTE: These changes were remove in 1.9.0

In 1.8.0, support for multiple huey instances was added (with thanks to @Sebubu and @MarcoGlauser for the patches). Although existing Django/Huey apps should continue to work, there is a new configuration format available and I'd recommend that you take a look at the docs and switch over to it:

Django integration documentation

v1.7.0

Backwards-incompatible change

Previous versions of huey would store the traceback and associated metadata for a failed task within the result_store, regardless of whether store_errors was true or not. As of 1.7.0, task exceptions will only be stored in the result store if store_errors is True. See #290 for discussion.

View commits

v1.6.1

  • Add backwards-compatibility to queue serialization protocol so that 1.6 consumers can continue to work with tasks enqueued by huey versions 1.5 and lower.

View commits

v1.6.0

  • Support for task pipelining and task function partials (which is not compatible with 1.5's task serialization format see note below).
  • Support for triggering task retries using RetryTask exception.
  • Support for task locking, restricting concurrency of a given task.
  • Getting result of task that failed with an exception results in a TaskException being raised.
  • Updated health check to ensure the task scheduler is always running.
  • Refactor implementation of task() and periodic_task() decorators, which should have the added benefit of making them easier to extend.
  • Refactored result-store APIs to simplify serialization / deserialization logic.
  • Fixed bug in serialization of task exceptions.
  • Added simple client/server implementation for testing locally. Blog post on the subject.

Task serialization format

In v1.6.0, the serialization format of tasks has changed to accomodate an extra piece of metadata. As a result, tasks enqueued with huey versions previous to 1.6 will not be able to be consumed by the 1.6 consumer.

At present there is a workaround available in 1.6.1, but it will be removed when 1.7.0 is released later.

View commits

v1.5.6

  • Allow arbitrary settings to be specified in task() decorators.
  • New task name format includes function module as part of task name.
  • Fix for operating systems that do not implement SIGHUP.
  • Fix bug in contrib.minimal task scheduler timing.

View commits

v1.5.5

  • Implemented pre-execute and post-execute hooks.
  • Implemented task cancellation mechanism as part of pre-execute hooks.

View commits

v1.5.4

  • Implemented atomic "set if not exists" for Redis and SQLite, which is used by the locking APIs.

View commits

v1.5.3

  • Includes addition of TaskLock and Huey.lock_task() helpers.
  • Extend Huey API to add method for creating the consumer.

View commits

v1.5.2

  • Added support for gracefully restarting the consumer using SIGHUP.
  • Fixed a bug where periodic tasks were not being given unique task IDs when executed by the consumer. Periodic tasks now receive a unique ID each time they are invoked.

View commits

v1.5.1

Added support for specifying a retry and retry_delay on periodic tasks. Simply pass the desired values into the periodic_task() decorator after the validation function, as keyword arguments.

View commits

v1.5.0

  • Allow all instances of a task to be revoked/restored by adding the revoke(), restore() and is_revoked() methods to all decorated tasks (where previously they were only available on periodic tasks).
  • Periodic task instances now have a unique identifier.
  • Added documentation on how to correctly use the Django consumer management command with the gevent worker model.
  • Logging will lazily resolve log messages.
  • Bug was fixed that prevented the local (non-global) task registry from working as intended. This is now fixed.
  • Docstrings added to the BaseStorage APIs.

Thanks to @mindojo-victor and @nachtmaar for help with some of the above items.

View commits

v1.4.1

  • Support using 7 to represent Sunday when doing day-of-week calculations in the crontab helper.
  • Fix bug #243, wherein Django interpreted boolean CLI arguments as having a boolean default value.

View commits

v1.4.0

Fixed a subtle bug in the way Huey calculated when to run the periodic task scheduler. If you had configured the consumer to check the schedule at an interval that was not a factor of 60, then there is a chance that periodic tasks may be scheduled at incorrect intervals from one minute to the next. This is fixed in 1.4.0.

Added better signal handling in order to support graceful shutdown. Graceful shutdown involves letting workers finish executing any tasks they may be processing at the time the shutdown signal is received. The default behavior is to interrupt the workers mid-task. Huey uses SIGTERM to shutdown the consumer immediately, and SIGINT to gracefully shutdown.

Added support for using either a global task registry, or a registry bound to a particular Huey instance. The default behavior is to use a global registry (backwards-compatible). To bind the registry to a single Huey instance, pass global_registry=False when initializing your Huey object.

Added a reschedule() method to the TaskResultWrapper.

Documentation clean-ups and additions, particularly around the logic used to handle datetime conversion. Also added docs on shutdown modes for huey consumer.

View commits

v1.3.1

Smarter conversion between datetimes, so that huey will correctly interpret naive or timezone-aware datetimes and properly convert to UTC when configured to do so. Previously, huey only operated on naive datetimes. Many thanks to @Antoine for this patch-set.

Documentation clean-ups and additions.

View commits

v1.3.0

Adds flag to preserve result-store value in certain circumstances. Contains yet more hacking at the consumer configuration options, specifically hard-coded defaults are removed from the option definitions.

The run_huey management command was simplified as we are dropping support for older (officially unsupported) versions of Django.

Added a sqlitedb contrib module that uses a local SQLite database instead of Redis for Queue persistence, task scheduling and result-storage.

View commits

v1.2.3

Contains an attempt at fixing the django management command handling of the default option.

View commits

v1.2.2

Contains small bugfix for an earlier bugfix meant to prevent time.sleep() from being called with a negative time interval.

View commits

v1.2.0

Removed the metadata APIs added in 1.1.0, as they seemed poorly-designed and altogether a decent idea terribly implemented. Perhaps something I'll revisit, but which should be easy to implement as a third-party library using the events APIs.

  • AsyncData is renamed to TaskResultWrapper.
  • Huey.result() is a new method that provides the result of a task, given a task ID.
  • Fixed a handful of bugs related to the error serialization.
  • Change the default consumer log handler from RotatingFileHandler to the vanilla FileHandler class.

View commits

v1.1.2

I've added a new API for fetching a task's result given on the task's ID. You can now call huey.result(task_id) and retrieve the result if the task has finished executing. Additionally, the Huey.result method accepts the same parameters as AsyncData.get, allowing you to block for results, specify a timeout, etc.

There is also a new parameter on the above methods, preserve=False. By default, the result store will delete a task result once it has been read. Specifying preserve=True ensures the data is not removed.

View commits

v1.1.1

This is a small release with a couple minor bugfixes.

  • Fixed task metadata serialization bug. #140
  • Small cleanup to event iterator storage implementation.
  • Updated getting started documentation to reflect changes in the 1.x APIs.

View commits

v1.1.0

  • Big changes to simplify the way Huey is instantiated. No changes should be necessary if already using RedisHuey.
  • Refactored the storage APIs and simplified the public interface. There is now a single object, whereas before there were 4 components (queue, result store, scheduler and event emitter).
  • Added methods for retrieving and introspecting the pending task queue, the schedule, results, and errors.
  • Errors can now be stored, in addition to regular task results.
  • Added metadata methods for tracking task execution, errors, task duration, and more. These will be the building blocks for tools to provide some insight into the inner-workings of your consumers and producers.
  • Many new events are emitted by the consumer, and some have parameters. These are documented here.

v1.0.0

What follows is a description of the changes between 0.4.9 and 1.0.0. There are some backwards-incompatible changes to be aware of as well as new options for the consumer. Most APIs are the same, however.

Backwards incompatible changes:

  • huey.djhuey moved to huey.contrib.djhuey. You will need to update any import statements as well as your Django INSTALLED_APPS setting to reflect the new module path.
  • Redis backend is now the only one available, and the corresponding code moved from huey.backends.redis_backend to huey.storage.
  • Removed the "RabbitMQ" and "SQLite" queue backends.
  • Removed the -t and --threads option from the consumer. You should now use -w or --workers.
  • Removed the -p and --periodic no-op options from the consumer. These are enabled by default so the option had no meaning.
  • The scheduler-interval option is configured using -s when previously it was -S. Furthermore, this must be a value between 1 and 60.
  • Removed the peewee_helpers module.

New features:

  • The queue consumer now supports multi-process or multi-greenlet execution models (in addition to multi-threaded, which previously was the only option).
  • Added pending(), scheduled() and all_results() methods to the Huey class to allow introspection of the Queue's state at the current moment in time.