> That cleanup can be explicit when needed by using context managers.
It certainly can be, but if a large part of the Python code you are writing involves native objects exposed through bindings then using context managers everywhere results in an incredible mess.
> Mixing resource handling with object lifetime is a bad design choice
It is a choice made successfully by a number of other high-performance languages/runtimes. Unfortunately for Python-the-language, so much of the utility of Python-the-ecosystem depends on components written in those languages (unlike, for example, JVM or CLR languages where the runtime is usually fast enough to require a fairly small portion of non-managed code).
The problem is the million small paper cuts. The stdlib changes are not all in pure python, many have implications for compiled modules like _ssl. The interpreter changes, especially compatibility with small interpreter changes that are reflected in the dis module, also require work to figure out
- The pure python repl started off in PyPy, although a lot of work was done to make it ready for prime time by the COython core devs
- The lessons from HPy are slowly making their way into CPython, see https://github.com/py-ni
- There were many fruitful interactions in fixing subtle bugs in CPython that stemmed from testing the stdlib on an alternative implementation
Third party libraries like SciPy scikit-learn, pandas, tensorflow and pytorch have been critical to python’s success. Since CPython is written in C and exposes a nice C API, those libraries can leverage it to quickly move from (slow) python to (fast) C/C++, hitting an optimum between speed of development and speed of runtime.
PyPy’s alternative, CFFI, was not attractive enough for the big players to adopt. And HPy, another alternative that would have played better with Cython and friends came too late in the game, by that time PyPy development had lost momentum.
Yes. The C API those libraries use is a good fit to CPython, a bad fit to PyPy. Hence CFFI and HPy. Actually, many if the lessons from HPy are making their way into CPython since their JIT and speedups face the same problems as PyPy. See https://github.com/py-ni
Sorry can you explain more the connection between PyPy and CFFI (which generates compiled extension modules to wrap an existing C library)? I have never used PyPy, but I use CFFI all the time (to wrap C libraries unrelated to Python so that I can use them from Python)
CFFI is fast on PyPy. The JIT still cannot peer into the compiled C/C++ code, but it can generate efficient interface code since there is a dedicated _cffi_backend module built into PyPy. Originally that was the motivation for the PyPy developers to create CFFI.
Thank you for the background info, and sorry for me explaining CFFI (I just wanted to be sure we were talking about the same thing). Being ignorant about PyPy, I honestly had no idea until now that there was a personnel or purpose overlap between CFFI and PyPy. I am very grateful for CFFI (though I only use it API mode).
PyPy core dev here. If anyone is interested in helping out, either financially or with coding, we can be reached various ways. See https://pypy.org/contact.html
Thanks, I should add PyPy to the list of projects I send a little to ... PyPy should be better supported by organisations + not need individual contributions, but things are where they are I guess.
Donated. Thank you and everyone else on the PyPy team.
I use PyPy regularly on an app of mine, and very often when I need to do some compute heavy load. Typically over 5x faster than CPython. It makes some stuff that takes impossibly long with CPython (nobody wants to wait 5 minutes...), to returning a response in a few seconds.
Another suggestion to add for you all (IDK how helpful.) When I see PyPy I see that its speed is faster for CPU-bound work but I'm thinking there is also I/O bound work that would see significant increases in the load they can handle. You could host a page that benchmarks common tasks like HTTP req/s (different types) with asyncio vs CPython. Could even have an automated tool that allows projects to benchmark performance from a web-page using PyPi without having to install or measure anything.
It would also be nice to see benchmarks of how much faster PyPy is getting each version. I know there is a tracking page but it tracks dozens of tests and has no absolute reference summary by version.
An easy chart to show v3.x is 10% faster than the last version would be great.
> Imagine how Open Source Software could improve if a consortium of nations put their money and resources into commissioning bug fixes and enhancements, which would be of collective benefit.
This is the business model of Quansight Labs, whose employees help maintain much of the scientific python stack. Mostly tech companies, not governments, sponsoring the work
It sounds like their point is that there's precedent for things in early stages only being provided as source, with binary releases coming later when things are more stable
reply