Meta is dedicating 3 engineers to get the nogil patches into cpython. There are some other companies stepping up as well. This is huge this is the closest we have ever been to solving the issue of the GIL.

  • teraflopsweat@lemm.ee
    link
    fedilink
    arrow-up
    7
    ·
    1 year ago

    As a filthy casual, could anyone give me a link or brief summary as to why the GIL should/shouldn’t go away?

    • mvirts@lemmy.world
      link
      fedilink
      arrow-up
      7
      ·
      1 year ago

      Basically there is no parallel execution of python code within a single process as long as the global interpreter lock exists. It prevents more than one python thread from running. However, many of the major libraries used in python call out to libraries that can and do release the gil to run native code in parallel.

      • Kale@lemmy.zip
        link
        fedilink
        English
        arrow-up
        5
        ·
        edit-2
        1 year ago

        The GIL only executes one thread at a time. A python program can be multithreaded, but only only thread runs in CPython at a time. If one thread does a system call (like copying a file), then when the python thread is sleeping, the system call can still run in the OS, so there are situations where multithreading can speed up Python programs, even running one thread at a time.

        You can run multiple instances of CPython, which is called multiprocessing, and each instance will run one python thread at a time. With different memory space, so all process communication has to be handled manually (afaik, by definition, threads share the same memory space, processes do not).

        Any library calls not written in Python don’t run in the interpreter, so most common critical things aren’t limited too badly. For example, I install a NumPy and SciPy library which are compiled against Intel’s MKL library. Any NumPy operations execute in MKL, not the Python interpreter, so are almost as fast as writing the program in C and compiling against MKL myself. And I can write Python and NumPy code about 10x faster than C/MKL. And if I’m on a computer that doesn’t have MKL, I can install a different NumPy library and it will execute just fine without changing the code.

        There’s a book called “high performance Python” that helped me figure out a lot of this.

        Edit: thought I was posting on the grandparents post instead of the parent post. Sorry.

    • anonono@lemmy.world
      link
      fedilink
      arrow-up
      3
      ·
      1 year ago

      a simple explanation but not 100% correct is that even if your code is made to run in parallel using threads, it will never use more than 1 core in your computer.

      getting rid of the GIL will let it use all the cores in the processor.

      the multiprocessing module “solved” this problem by forking processes instead of threads, but it’s not ideal for a lot of workloads.

    • Dogeek@sh.itjust.works
      link
      fedilink
      arrow-up
      2
      ·
      1 year ago

      The GIL is a thread lock. It prevents threads from accessing the same memory space, eliminating race conditions and more importantly, keeping the reference counters correct so that the python garbage collector can correctly free memory (avoiding leaks)

  • verdare [he/him]@beehaw.org
    link
    fedilink
    English
    arrow-up
    5
    ·
    1 year ago

    For the amount of multiprocess code I’ve written in Python, I’m embarrassed that this is the first I’m hearing about the noGIL build. Really excited to see this feature become the default, though it looks like it will take a while. GIL has definitely been a pain in the ass for fully utilizing multi-core systems.

    • Rev@ihax0r.comOP
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      Like cython?

      Its a whole lot of magic. I many times have to look at the generated c code because its doing something weird especially when generating c++

      • Zeth0s@lemmy.world
        link
        fedilink
        arrow-up
        1
        ·
        1 year ago

        Sorry I removed the old message because I thought was not interesting.

        I mean a compiler python to binaries, focused on performance optimization.

        Currently with cython you compile to c and then compile to binaries.

        Python ecosystem has become so complex and so widely used that a dedicated compiler would have its uses, and could do a lot in further optimizing performances