Skip to content

Merge dev: security fixes, protocol-user bug fixes, Python 3.10 support - #110

Merged
JE-Chen merged 77 commits into
mainfrom
dev
Sep 23, 2026
Merged

JE-Chen merged 77 commits into
mainfrom
dev

Conversation

@JE-Chen

@JE-Chen JE-Chen commented Sep 23, 2026

Copy link
Copy Markdown
Member

Merges 56 commits from dev. Details for each item are in docs/updates/2026-09.md.

Security and dependencies

  • Clears the 52 Dependabot alerts: the mitmproxy extra is gone, the floors are raised (kafka-python 2.3.2, cryptography 50), and uv.lock is re-locked (U-20260923-01, U-20260923-20).
  • The executor registers only the SAFE_BUILTINS allowlist.
  • The PII scrubber runs in linear time and labels card numbers correctly (U-20260923-16).

Fixes

  • Python 3.10: the package imports again. tomllib is loaded lazily with a tomli fallback (U-20260923-21).
  • Protocol users: nine bugs are fixed (U-20260923-19), including Modbus on pymodbus 3.15, APNs rejections, Redis/SMTP/IMAP/SQL error handling and response sizes, scenario weight 0, and false-like conditions.
  • The asyncio engine counts 4xx/5xx as failures; reports keep request order; Excel cell references and PDF titles are fixed; Lambda/Cloud Run launcher fixes; find_breaking_point probes the count it settles on; test-catalog tag parsing; DB fixtures dispose their engines.
  • The log file moves out of the working directory.

Tests: 372 → 1045 passed. New suites cover the protocol templates and the scenario, data, dx, governance and ai helpers, plus a check of the counts quoted in the docs.

Behaviour changes to note

  • find_breaking_point returns safe_users=0 when no probed count passes.
  • An APNs reply other than 200 fails the step.
  • Scenario weight: 0 excludes the task, and a False variable in run_if is false.

Work sat uncommitted for weeks in several repositories of this workspace
and progress files grew into change logs. The project guidelines now ask
for a commit at every stage, keep progress.md to open items only, record
finished work in monthly batches under docs/updates/ with an index and
query commands, and add a short architecture.md overview.
Two parallel sessions picked the same entry number because claiming an ID
first does not stop simultaneous writers. Entries are now numbered under
an atomic mkdir lock in docs/updates/, with a duplicate check before commit.
Add user templates and proxies for HTTP/3, GraphQL over WebSocket,
AMQP, NATS, Pulsar, CoAP, Cassandra, Elasticsearch, SMTP, IMAP, FTP,
SFTP, fuzzing HTTP, SOAP, LDAP, SNMP, Modbus, OPC UA, ZeroMQ, Thrift,
Memcached, Neo4j, Couchbase, etcd, Consul, Vault, Web Push, APNs and
FCM, plus the shared _common and _protocol_base helpers, and register
them in LocustUserProxy and _USER_REGISTRY.
Export the new modules from je_load_density, register their LD_*
commands in the executor, add the bench and shell subcommands and the
generate_from_openapi / generate_from_curls MCP tools, declare the
optional-dependency extras in pyproject.toml, and add the smoke tests.
architecture.md drops its uncommitted markers; progress.md and
docs/updates record the staged commit.
Raise the floors the code can reach (kafka-python>=2.3.2, cryptography>=48.0.1,
mcp>=1.28.1), cap mcp below 2.0 whose low-level Server dropped the decorators
the MCP server uses, drop the mitmproxy extra that pinned cryptography, h2 and
msgpack below their fixes, re-resolve uv.lock and move @vscode/vsce to 3.9.2.
Importing the package imports locust, whose gevent patch_all() stalls the
mcp SDK's thread-based stdin reader, so the server never answered; mcp 2.x
also dropped the decorators it used. Serve initialize, ping, tools/list and
tools/call directly, keep the protocol stream clean by moving other stdout
writes to stderr, and empty the mcp extra.
pymodbus 3.15 renamed slave to device_id, so every read and write raised
TypeError there; the keyword now follows the installed signature.
pip-audit flags PYSEC-2026-3552, fixed in cryptography 50.0.0; the lock
already resolves 50.0.1.
toml_file imported tomllib at module level, which 3.10 does not have, so
importing je_load_density failed there although requires-python allows
3.10. Load tomllib lazily, fall back to the tomli backport and declare it
for Python < 3.11. The YAML smoke test now skips when pyyaml is missing.
An entity bomb raised defusedxml's EntitiesForbidden past the parser's
ParseError handler; the XML and JSON helpers also dropped the original
error when re-raising.
@codacy-production

codacy-production Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 2990 complexity · 397 duplication

Metric Results
Complexity 2990
Duplication 397

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

generate_from_openapi passed base_url to openapi_to_tasks, which had no
such parameter, so the function, LD_generate_from_openapi and the MCP
tool raised TypeError on every call. base_url now replaces the spec's
first server URL.
The MCP tools created, read and wrote whatever paths the client sent, and
the client is a language model that can be steered by what it reads.
Paths must now resolve inside JE_LOAD_DENSITY_MCP_ROOT (default: the
working directory).
The Helm pods get ephemeral-storage requests and limits and no
service-account token (they never call the Kubernetes API); the
operator gets CPU, memory and storage requests and limits, and its
image runs as a non-root user. helm lint and kubeconform pass.
socket, SSE, WebSocket and MQTT stored the setter's host and never read
it; SMTP, SNMP, SOAP, Thrift, Vault, WebPush and ZMQ did the same with
connection. host is now the user's default target and connection
supplies default step fields; a value in the step still wins.
…m in CI

main.tf used set blocks, which helm provider 3.x (what the unpinned
range resolved to) rejects; it now passes one values document and pins
both providers below 4.0. The operator image installed je_load_density
[all] it never imports and the non-root user step collided with Debian's
existing 'operator' account; it now installs kopf and kubernetes from a
hash-locked, wheels-only requirements file. A deploy-check workflow runs
helm lint, kubeconform, terraform validate, the image build and a parse of
every CI template.
The metadata IPs and http:// URLs are the payloads an SSRF probe sends,
and plain FTP is the protocol the FTP user load-tests.
Adds settings.gradle.kts, enables dependency locking for every
configuration and commits the gradle.lockfile Gradle 8.10.2 generated.
The plugin builds with gradle buildPlugin.
…yloads

Third-party actions are pinned by commit SHA, the kubeconform tarball is
checked against its release checksum and curl only follows HTTPS
redirects, pyyaml comes from a hash-locked file, and the remaining
loopback SSRF payloads carry their NOSONAR reason.
check_excessive_data_exposure only looked for the quoted name inside
string values, so a parsed body such as {"password": "x"} was never
flagged. Adds tests for the OWASP checks, SSRF targets and metadata-leak
detection, and the rate-limit probe with urlopen stubbed.
… mutation

mutate_json returned a single copy for an empty dict whatever count was,
so expand_task_fuzz raised IndexError on a task with json={}; it now
returns count copies. _mutate_string listed the reversed seed twice and
now keeps only distinct variants. Adds shape tests for the fuzz,
GraphQL, JWT and smuggling payload generators.
The manifest named an icon.png that does not exist, so Chrome refused to
load the extension; and a Manifest V3 service worker has no
URL.createObjectURL, so stopping a recording threw instead of
downloading. The HAR now goes out as a data URL. Also declares the chrome
global for ESLint and names the <all_urls> filter once.
…tor pod

Both templates pasted their parameters into a shell script as text, so a
value with quotes or ';' became part of the command. The operator
container now runs non-root with no privilege escalation, a read-only
root filesystem, no capabilities and the runtime seccomp profile.
Log the errors three helpers swallowed, escape Excel and PDF text with
html.escape instead of importing xml.sax, drop unused imports, annotate
the intended FTP use, placeholder JWT secret and fixed-argument
subprocess calls, and give a timing-sensitive smoke test a wider window.
Codacy's Semgrep rule id did not match the partial id in the earlier
comments.
@sonarqubecloud

Copy link
Copy Markdown

@JE-Chen
JE-Chen merged commit 0d21e3f into main Sep 23, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant