Skip to content

Normalize dot segments in URL paths per RFC 3986, including percent-encoded forms - #3854

Open
waw666waw666 wants to merge 1 commit into
encode:masterfrom
waw666waw666:fix/normalize-path-dot-segments
Open

waw666waw666 wants to merge 1 commit into
encode:masterfrom
waw666waw666:fix/normalize-path-dot-segments

Conversation

@waw666waw666

Copy link
Copy Markdown

Summary

URL path normalization deviates from RFC 3986 in two ways:

  • A trailing . or .. segment is dropped instead of being replaced with a trailing /. For example httpx.URL("https://example.com/a/b/..") currently gives https://example.com/a, where the remove_dot_segments algorithm in RFC 3986 §5.2.4 gives https://example.com/a/.
  • Percent-encoded dot segments are not recognized as dot segments at all. httpx.URL("https://example.com/a/%2e%2e/b") keeps the path as-is, even though %2E percent-decodes to the unreserved character ., and RFC 3986 §6.2.2.2 requires such octets to be decoded before syntax-based normalization. This also makes the raw and decoded views of the same URL disagree: the .path of that URL is "/a/../b".

Root cause

normalize_path() in httpx/_urlparse.py only matches the literal components "." and "..", and its fast path returns early for any path without a literal ".", which includes every percent-encoded form. It also has no handling for the trailing-segment case in the RFC algorithm, where a final dot segment is replaced by "/".

Change

  • Add a _dot_segment_length() helper that classifies a path component as a "." or ".." segment after decoding %2e sequences (case-insensitive). Only components that are exactly dot segments match, so segments such as foo%2ebar or ... are left untouched.
  • Use it in normalize_path(), both for the fast-path check and for segment processing.
  • Replace a trailing dot segment with a trailing / instead of dropping it.

Relative references are unaffected, as normalize_path() is only applied to URLs with a scheme or authority.

Tests

Added regression tests in tests/models/test_url.py:

  • test_url_normalized_path_trailing_dot_segments — trailing ./.. cases, including the RFC 3986 §5.2.4 example /a/b/c/./../../g → /a/g.
  • test_url_normalized_path_percent_encoded_dot_segments — %2e, %2E%2E, %2e., .%2E forms, plus non-dot segments containing %2e that must be preserved.

All 13 new cases fail or pass as expected: the 8 previously-wrong cases fail on master and pass with this change. The full tests/models/ suite passes, and ruff format, ruff check and mypy are clean.

Checklist

  • I understand that this PR may be closed in case there was no previous discussion. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly. (No documentation changes needed: this aligns the implementation with the RFC behaviour the docstrings already describe.)

Treat percent-encoded dot segments (%2e forms) as dot segments during
path normalization, and replace a trailing dot segment with a trailing
slash instead of dropping it, as required by RFC 3986 sections 5.2.4
and 6.2.2.2.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant