Sure, but maybe upgrading the existing Python 2 code wasn't the problem: the vast majority of the Python 2 code alive when Python 3 was released was probably scrapped by the time Python 2 was EOL'd. Over a decade is a very long time for a codebase to live. (Again, there are exceptions, but the vast majority of websites or data processing scripts or whatnot are significantly re-written or decommissioned in – pulling a number out of my arse – five years or so.)
The (comparatively) few Python 2 projects that were alive by the release of Python 3 and still alive by the EOL of Python 2 could probably have been upgraded manually, rather than with a tool.
On the other hand, people not wanting to write new code in Python 3 because it would be incompatible with the Python 2 ecosystem – that was a real problem. That problem could have been prevented with a 3to2 tool, which would compile the Python 3 code to Python 2, allowing it to be used with the existing ecosystem.
If only people would have written new code in Python 3 right away, we would have been in a far better position by the time Python 2 would have been EOL'd, purely due to natural turnaround of code bases.
Porting Python 2 code to Python 3 was never the problem. Writing new code in Python 3 was.
> Porting Python 2 code to Python 3 was never the problem.
You're, frankly, high as a kite. As the author of one such port on a large codebase, porting Python 2 code to Python 3 was absolutely a problem.
The issue with 2to3 is that it assumed you'd do the conversion as a one-shot and straight flip over with entirely separate codebases (or dropping Python 2 entirely) from there on, which is completely insane.
What the vast majority of transitioning projects needed was a way to migrate progressively through cross-version codebases:
* libraries weren't going to drop older Python versions, the very few which attempted that (I'm aware of dateutil at least) were extremely painful to users and IIRC ended up rolling back to a more progressive codebase
* programs being distributed were not going to drop Python 2 until EOF at least
* large codebases simply could not afford to perform a completely untested and uncheckable one-shot transition
That's why the community created cross-version support tools like six and friends, and lobbied hard for the reintroduction of cross-version features into Python 3 e.g. reintroduction of `callable` (3.2), reintroduction of the "u" prefix (3.3), reintroduction of % on bytes (3.5) and I'm sure others.
> Again, there are exceptions, but the vast majority of websites or data processing scripts or whatnot are significantly re-written or decommissioned in – pulling a number out of my arse – five years or so. […] Python 3 because it would be incompatible with the Python 2 ecosystem
What do you think "the python 2 ecosystem is" except for a ton of projects which needed to be ported from Python 2 to Python 3 exactly? "the vast majority of websites or data processing scripts" is not an ecosystem, tooling and libraries are.
Do you somehow think numpy and lxml and django just scrapped their entire Python 2 codebase and rewrote the entire thing from scratch? Of course not.
> That problem could have been prevented with a 3to2 tool, which would compile the Python 3 code to Python 2, allowing it to be used with the existing ecosystem.
3to2 would not have worked any better better than 2to3. The syntactic incompatibilities between the version were not the hard part, the semantic ones were, and 3to2 would not have succeeded any better than 2to3 there because it simply couldn't.
All 3to2 would have given you is a broken Python 2 codebase from a Python 3 codebase you could not even run.
Isn't "migrating progressively through cross-version codebases" exactly what a 3to2 compiler would allow you to do? You'd write new code (and incrementally port old code) to Python 3, and then when you build the project you'd compile the Python 3 code down to Python 2 before building the entire project as Python 2, including all dependencies.
So for over a decade, you'd be writing Python 3 code but building a Python 2 project. After a decade, one would hopefully have managed to port everything over to Python 3, and can then drop the "compile to Python 2" step.
A 3to2 tool would solve exactly the problems you're bringing up, with no intoxication needed!
----
I don't see why 3to2 would not have worked. It's a compiler. We have written compilers before. We can write compilers. Compilers work. Both languages are Turing complete and about the same in expressiveness, so clearly compiling one to the other is something we can do.
What we cannot (reasonably) do is produce nice, human-looking target code from a compiler. This is what 2to3 attempted to do, and what a 3to2 would never need to do.
Who cares if the Python 2 code it produced looked a little wonky, as long as it was correct and roughly recogniseable as being built from the original Python 3 code? The Python 3 code is the one humans write and edit. The Python 2 code is just compiler output to make it compatible with all of the Python 2 ecosystem until one is ready to go fully Python 3.
----
This is also not something completely foreign and unheard of. This is exactly how I've been porting legacy JavaScript applications to a more modern ECMAScript approach: write new code in the modern way, compile it down to legacy code compatible with the existing legacy code, and then run that.
Slowly but surely, more and more things are being written in the modern fashion, and eventually, when everything has caught up, the compatibility conversion to the legacy code can be dropped.
> Isn't "migrating progressively through cross-version codebases" exactly what a 3to2 compiler would allow you to do?
No. You'd have to migrate everything to Python 3 at once, and then "maintain" two different and incompatible codebases, one generated from the other.
> A 3to2 tool would solve exactly the problems you're bringing up, with no intoxication needed!
It would half solve (at best) one of them.
> I don't see why 3to2 would not have worked. It's a compiler.
Unless your 3to2 would literally reimplement Python 3 in Python 2, it would be a translator / bridge. Which can't work due to the semantics difference between Python 2 and Python 3: not every change translates mechanically and reliably. If they did, 2to3 would have worked.
> Who cares if the Python 2 code it produced looked a little wonky,
The maintainer who has to debug, probably.
> as long as it was correct
Which it wouldn't be.
> This is also not something completely foreign and unheard of. This is exactly how I've been porting legacy JavaScript applications to a more modern ECMAScript approach: write new code in the modern way, compile it down to legacy code compatible with the existing legacy code, and then run that.
The "modern way" of javascript is additive, it's about adding new features to the language, some of which can be implemented in terms of the old one (and which you can thus "backport" through a compiler or extending the existing APIs).
You don't run into pieces of code which are syntactically identical but semantically divergent. Which you absolutely do in P2 v P3.
> You'd have to migrate everything to Python 3 at once, and then "maintain" two different and incompatible codebases, one generated from the other.
Absolutely not. Well, you could, but the better approach would be to write individual modules in Python 3, and then compile those down to Python 2 at which point they can talk to the existing Python 2 code freely.
> Unless your 3to2 would literally reimplement Python 3 in Python 2
That's exactly what I'm suggesting. Take the existing Python 3 compiler and swap out the backend to generate Python 2 code instead of bytecode.
This is stuff people do every day for other languages. It is very far from rocket science.
Amen. Babel gets a lot of things right (and arguably has a tougher job than a 3to2 compiler would, given the speed of change and the target breadth / problem surface area).
Babel is not limited to JavaScript frontends. You can take any language and write a Babel front-end for it. This is, for example, how Fable compiles F# code to JavaScript. Those two languages are very different from each other -- far more differences than Python 2 and Python 3 -- and it works brilliantly.
I made all my Py2 code forward compatible with 2to3. It was mostly a smooth process when you're using the future imports and know what landmines to avoid. It still required manual intervention at times with Unicode support and libraries that have breaking API changes between versions.
The end result is something that can be Py3 native whenever I want. 2to3 is the right tool for the job. There is no way to map all Py3 features back into Py2 so 3to2 would be a broken mess that only supported a subset of the language.
> There is no way to map all Py3 features back into Py2 so 3to2 would be a broken mess that only supported a subset of the language.
Oh but there is. If you can map them to bytecode, you can map them to Python 2, which is a fully capable language. It won't be a one-to-one map, but that's also not required.
good theory, doesn't align with my (limited) experience. writing new code to be both 2 and 3 compatible was quite possible and not too tricky. six helped, `from __future__` imports, etc. however, everywhere i've worked still had some python 2 only code running - even to this day unfortunately.
the real issue is 2to3 didn't work so well, or exposed the exact issues why python 3's breaking changes were needed (80% sloppy bytes/str/unicode handling).
