Route task logs to stdout through the worker log handler - #58
Merged
Merged
Conversation
The JSON handler was attached to the `multiprocessing` logger only, so records of any other logger never reached it: an `info()` call inside a task was dropped, and an `error()` call fell back to Python's lastResort handler, printing unformatted text to stderr instead of honoring `--log-format`. Configure logging per process instead: the handler goes on the root logger at INFO, replacing handlers Django installs in `django.setup()` and any application handler, and it writes to stdout. Every record from tasks, threadmill, and Django now uses the configured formatter.
Python 3.14 defaults to forkserver on Linux, and a worker forked from the long-lived forkserver inherits that server's stdout rather than the file descriptor the `capfd` fixture swaps in for the test. The records of the worker never reached `capfd`, so `test_run__routes_task_logs_to_stdout` failed on CI while it passed on macOS, where spawn is the default. Run the executor with the spawn start method for this test and restore the original method afterwards, the same way the neighbouring spawned-worker test does. The assertions are unchanged, and the test still fails with the handler attached to the multiprocessing logger only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A logger used inside a task never printed. The JSON handler was attached to the
multiprocessinglogger, so records from any other logger never reached a handler:logger.info(...)in a task was dropped outright, andlogger.error(...)fell back to Python'slastResorthandler, printing unformatted text to stderr and ignoring--log-format.Worker processes now own their logging configuration.
configure_logging()puts the shared handler on the root logger atINFO, writing to stdout, and replaces the handlers Django installs duringdjango.setup()plus any application handler. It runs inTaskExecutor.run()for the supervisor and inWorkerProcess.run()for each worker, afterdjango.setup(), so Django cannot reinstall its console handler afterwards.The result: records from tasks, threadmill, and Django all pass through the configured formatter (single-line JSON by default,
--log-formatto override) on stdout, with no duplicated or unformatted lines on stderr.Worth a look during review: because the worker owns logging, handlers configured in
LOGGINGare replaced inside the worker command. The alternative kept application handlers and duplicated everydjango.*record across two formats.Covered by unit tests for
configure_logging()(handler placement, foreign handler replacement, placeholder loggers) and an end-to-end test that runs a logging task and asserts its JSON record lands on stdout rather than stderr.