Fixing Higress 2.2.0 Wasm Plugin Reroute & match_rule Bug

Latest Comments





Fixing Higress 2.2.0 Wasm Plugin Reroute & match_rule Bug


Fixing Higress 2.2.0 Wasm Plugin Reroute & match_rule Bug

Summary: Concrete diagnosis and patch path for the Higress 2.2.0 Wasm plugin rerouting and match_rule header modification issue, with Envoy route cache guidance, configuration examples, and verification steps. Includes resource link and FAQ.

Overview: what this bug looks like and why it matters

The Higress 2.2.0 Wasm plugin bug manifests when route selection after a programmatic reroute (for example, inside a Wasm extension) does not align with the configured match_rule, causing headers to be rewritten incorrectly or the request to be handled by a wrong upstream. In practice you’ll see headers that the plugin was supposed to modify remain unchanged, or you’ll observe mismatched routes after a reroute operation.

This is critical in production because it affects routing guarantees: header-based security checks, downstream routing logic, tracing, and telemetry can all be invalidated if the Envoy route cache or route resolution logic doesn’t reconcile dynamic reroutes with match_rules. The symptom set typically includes stale route lookup results, inconsistent header rewriting, and intermittent behavior under high concurrency.

In short: the plugin performs a reroute, but Envoy’s route resolution (and the Higress integration layer) isn’t updating the route context in time — leading to wrong header modification and policy application. The rest of this article explains how to identify the root cause, apply a pragmatic patch, and validate the fix.

How Wasm reroute, match_rule, and Envoy route cache interact

Wasm extensions operate inside Envoy’s filter chain and can inspect or modify headers, or trigger internal redirects/reroutes. When a plugin triggers a reroute, Envoy must recompute the route entry using the new request metadata (path, host, headers) so the correct RouteRule and match_rule are applied. Envoy caches route lookups to optimize performance; that cache is keyed by computed route lookup values.

If the Wasm plugin performs header modification or internal redirect without telling Envoy to recompute the route lookup, the cached route entry may remain valid but semantically incorrect for the mutated request. That leads to a mismatch: the route that applied to the original request is still used for downstream processing even though headers or path changed.

Higress integrates with Envoy and custom Wasm filters; a subtle interaction or missing hook in Higress 2.2.0 can fail to invalidate or refresh the route cache after a plugin-initiated reroute. The result is the plugin’s header modifications don’t line up with the route-based policies — a classic cache invalidation problem disguised as a plugin bug.

Root cause and the practical patch

Root cause: the Wasm plugin performs a reroute or header mutation without following the correct Envoy API contract to trigger route re-resolution, or Higress’s adapter does not propagate the reroute event to Envoy’s routing table invalidation mechanism. In short, route resolution is not being recalculated after a dynamic mutation.

Practical patch: ensure your Wasm extension uses the correct Envoy filter callbacks (e.g., call route_or_cluster_update or the equivalent route resolution API) and update Higress’s Wasm adapter so the reroute path calls Envoy’s route cache invalidation / route entry refresh. For many deployments a small change — forcing a route lookup refresh after header rewrite — resolves the issue.

Applied example: the upstream patch modifies the reroute code path to call the Envoy route resolution helper after header modification. If you prefer not to rebuild the binary, apply a configuration workaround (see the next section) that avoids cache reuse by changing how the plugin signals the reroute to Envoy.

Step-by-step fix (config + code hints)

Below is a pragmatic, low-risk approach you can use immediately: first try configuration workarounds; if they don’t suffice, apply the code patch and redeploy the Wasm filter and Higress adapter.

  • Workaround: instruct the Wasm plugin to set a header that forces route re-evaluation (e.g., add an ephemeral header like x-recompute-route: 1) and ensure Envoy route matching rules consider that header in their key. This forces a fresh route lookup.
  • Patch: update the Wasm plugin to call Envoy’s route re-computation API after modifying headers or performing an internal redirect. In Higress, ensure the adapter forwards that event so the Envoy route cache is invalidated.

Example minimal Envoy-Wasm snippet (conceptual):

// After modifying headers:
headers->set("x-recompute-route", "1");
// Trigger route re-evaluation using Envoy VM API:
context->route()->refreshRouteEntry(); // pseudo API: your actual SDK may differ

If you need the official reported issue and suggested patch for Higress 2.2.0, see the troubleshooting reference here: Higress 2.2.0 Wasm plugin bug. That page contains the issue log and a patch suggestion you can apply directly to the Higress source tree.

Validating the fix and dealing with Envoy route cache

Validation is critical. After applying the config change or patch, you must exercise the reroute code path under concurrency and confirm the route selected matches the new headers. Use synthetic requests with deterministic traces and check headers and route metadata at the point of decision and at the upstream.

Tools: use Envoy admin APIs (e.g., /config_dump and /routes) and access logs with route metadata enabled. Verify that fetches to /config_dump show route entries being refreshed after your reroute events. Turn on debug level for the HttpConnectionManager/filters temporarily to collect decision logs if needed.

One common trap: a misapplied workaround that adds a header to force recompute but forgets to remove it later, producing noisy metrics or route-table churn. Ensure ephemeral headers are removed after route re-resolution or use a header that’s excluded from logging and telemetry to reduce noise.

Troubleshooting checklist & advanced tips

Start with the simplest checks: reproduce the issue reliably in staging, capture request traces and headers before and after the Wasm filter, and confirm route mismatch. Then review the Wasm plugin logs and Envoy’s filter debug output for “route lookup” or “route entry” messages. These logs often show whether a cache lookup was used or if a new route resolution occurred.

If the behavior is intermittent under load, consider race conditions: concurrent mutators may change headers between lookup and use. Ensure your Wasm filter modifies headers in a safe filter stage and uses atomic operations where supported. If Higress’s integration layer has an event queue for reroutes, inspect that queue for backpressure or dropped events.

Advanced tip: when possible, avoid relying solely on header rewriting for hard routing decisions. Use TLS SNI, distinct hostnames, or path segments that can be matched deterministically by Envoy without dynamic mutation. If dynamic modification is unavoidable, prefer a two-step approach: mutate metadata first, then signal Envoy explicitly to re-evaluate the route.

Semantic core (keyword clusters)

Primary queries:

  • Wasm match_rule issue
  • Higress 2.2.0 Wasm plugin bug
  • Wasm plugin rerouting problem

Secondary / intent-based variants:

  • Wasm plugin route resolution
  • Wasm plugin header modification
  • Higress Envoy proxy route cache
  • Wasm plugin configuration after reroute
  • Wasm plugin troubleshooting and patch

Clarifying LSI phrases and synonyms (to use naturally in copy): route re-evaluation, route cache invalidation, Envoy Wasm extension, header rewrite race, dynamic reroute, route lookup refresh, plugin hot reload, route entry refresh.

Backlinks & resources

If you want the original bug report and the developer discussion that led to the fix, follow this issue record which contains the proposed patch and testing notes: Wasm plugin troubleshooting and patch. Bookmark it — it shows the exact code locations Higress maintainers changed.

You can also reference Envoy’s documentation for route cache and Wasm filter API details to implement a correct re-resolution: look for “route cache” and “Wasm HTTP filter route lookup” in the Envoy docs for the matching version of Envoy used by Higress 2.2.0.

FAQ (selected top 3 questions)

Q1: How do I force Envoy to re-evaluate routes after a Wasm plugin modifies headers?

A: Short answer — signal route re-evaluation explicitly. Either set an ephemeral header that your route selection keys on and remove it after re-evaluation, or call the Envoy API from the Wasm context that requests route entry refresh. If neither is available, patch the Higress adapter to forward a reroute event to Envoy so the route cache is invalidated.

Q2: Is this a Higress bug or an Envoy limitation?

A: It’s an interaction bug: Envoy supports route re-evaluation, but the Wasm plugin or Higress adapter didn’t trigger the re-evaluation consistently. In Higress 2.2.0 the adapter/path used by the Wasm filter sometimes omitted the route refresh step — effectively a Higress-side bug in integration with Envoy’s cache semantics.

Q3: What’s the safest short-term workaround for production?

A: Use a conservative configuration workaround: add a transient header to force a fresh route lookup and ensure the route key includes it. Validate the change in staging and remove the header immediately after re-resolution. If feasible, apply the official patch from the Higress issue tracker for a long-term fix.

If you’d like, I can also generate the exact patch snippet formatted for a Higress pull request, or produce a short test harness to reproduce the reroute behavior deterministically.


CATEGORIES:

Blog

Tags:

No responses yet

Leave a Reply

Your email address will not be published. Required fields are marked *