The method
A port has a reference: the code it replaces. So the question "is the port right?" can be asked of the machine, one input at a time. Run the old code and the new code on the same input and compare what comes back. Keep going until they disagree or the inputs run out.
We used equiv, our open-source checker, for most of this. It generates inputs, runs both sides in their own Python process, and reports the first input where they differ along with both results. Every run writes a receipt that reproduces byte for byte. It tests on generated inputs, so a clean run means no difference was found on those inputs. It does not prove the two are the same.
The pattern across all five projects was the same. A port's own tests run the inputs its authors thought of. The differences turned up on inputs nobody had written a test for: text that is not English, SQL from a different dialect, names with a slash in them.
django-rusty-templates
django-rusty-templates is a Rust implementation of Django's template engine that aims for identical output. We compared its filters with Django's own, 27 checks over 46,758 inputs. wordwrap panicked on text starting with a non-breaking space. center miscounted text that is not English. slugify kept a leading underscore that Django strips. render() returned a plain string where Django returns a safe one (#587).
A second pass on chained filters found three more. {{ 'a<b'|capfirst|length }} gave 6 where Django gives 3, because capfirst escaped too early. upper left a safe string unescaped, and escape escaped a safe string twice (#593).
The owner called the first set "great finds". We sent the fixes as five pull requests, and all five were reviewed and merged. The wordwrap fix is a port of Django's own wrapping and carries a property test that checks it against Django directly.
LiteLLM
LiteLLM moved parts of its core to Rust in 1.110. We compared the last Python release, 1.109.1, with the release candidate 1.110.0rc2, each in its own environment. Two things changed that were not meant to.
On the Anthropic, Vertex and Azure routes, temperature=1 was now refused for 72 of 405 Claude model names, where 1.109.1 accepted it (#51394). A contributor reproduced it, and a maintainer fixed it in #52341.
The Rust-backed stand-in for tiktoken.Encoding also differed from tiktoken in two edge cases: encode_with_unstable on a lone surrogate, and is_special_token on an ID outside 32 bits (#51601). The fix, #51720, checks both cases against tiktoken itself.
sqlfluff
sqlfluff ships a Rust parser next to its Python one, and its CI already runs both on the project's own fixtures. On those they agree: 16,442 statements. CI never runs one dialect's parser on another dialect's SQL. We did, and three parsers diverged.
- T-SQL read
DATEADD('year', day, month)differently in Rust and Python (#9095). Fixed by a maintainer the next day. - BigQuery recovered differently from
select a:from FROM t(#9096). Fixed two days after the report. - Snowflake built different trees for
MERGE ... WHEN NOT MATCHED THEN INSERT *(#9097). Another user reproduced it and traced the cause, and a contributor has taken on the fix.
ua-parser
uap-python can parse user agents with three interchangeable resolvers. They gave a different device for Applebot's published desktop user agent (#334). The maintainer traced it to a rule in the shared regex file and to a fallback in one resolver, and wrote the fix (uap-core#675). We ran all three resolvers over the 17,820 user agents in uap-core's tests. On main they disagree on 9. On the fix branch they disagree on none.
dulwich
dulwich, the pure-Python Git implementation, sorts tree entries in Rust with a Python fallback. We generated Git trees and compared the two. When entry names contain a slash, the Rust comparison is not a consistent order. Below 21 entries the two sort differently. From 21 entries the Rust sort panics, and the panic is a BaseException, so except Exception does not catch it. The maintainer reproduced it the day we reported it and merged a fix the same day (#2582). We re-ran our trees against the fix: no differences.
Check your own port
If a function has a Python version and a port, point equiv at both:
equiv review port.py original.py parse str --errors-agree
--errors-agree counts it as agreement when both sides raise, whatever the exception class, since a port rarely raises the same type. To compare two releases of an installed package, give each side its own interpreter:
equiv review new.py old.py fn str \
--reference-python old-env/bin/python \
--candidate-python new-env/bin/python \
--versions-of yourpackage
Each side reports which version it actually ran, and that goes into the receipt. equiv works on functions that take and return plain Python values. Its scope document says what it refuses.
If you are porting a library and want it checked against the original before your users find the gaps, we run parity audits. Write to [email protected].