Biology can use discrete methods; we have discrete neurons (and other cells), discrete chemical species like ATP, discrete amino acids, discrete genes made of discrete nucleotides, etc.
I dont get this. Just because you don't have the GIL doesnt mean that your previously single threaded code is now multithreaded and stepping on itself.
From previous discussions it's my understanding the c integration is going to be the cause for the issues.
From a python perspective it wouldn't necessarily be a big change, but everything can branch out to c, and there you're going to get in trouble with shared memory.
The most significant impact of this change is that python threads can run at the same time. Before, calls to C API were essentially cooperative multithreading yield points - even if you have 100 python threads, they can't run concurrently under the GIL. 99 are blocked waiting on the GIL.
C extensions have always been able to use real threading. But now there is no GIL to synchronize with the python interpreter on ingress/egress from the extension.
No, but it means that your previous multithreaded code is no longer automatically prevented from stepping all over itself by having multiple threads accessing the same data:
GIL removal doesn't mean "make all of this lockless" it means "replace the GIL with fine-grained locking". So those problems are still solved for Python code. The three issues are the amount of work it takes to do right, the performance cost for single-threaded code, and the CAPI.
I'm simultaneously scared and enlightened seeing all these comments acting as if the GIL is some magic "makes your code thread/concurrency safe" pancea. I always saw it as a shim/hack to make cpython specifically easier to implement, not something that inherently makes code more thread safe or atomic. It's just more work to do things "the right way" across application boundaries, but from my understanding this PEP is Meta commiting to do that work.
Removing the lock creates problems in existing code in practice. This is an ecosystem that has less focus on standards and more on "CPython is the reference implementation".
What non-transparent GIL specific behavior are developers relying on exactly?
When I say GIL specific behavior, I mean "python code that specifically requires a GLOBAL interpreter lock to function properly"
Not something that simply requires atomic access or any of the garuntees that the GIL has this far provided, but like, specifically code that requires GIL like behavior above any CPython implementation details that could be implemented with more fine grained concurrency assurances?
I've seen some really cursed python in my days, like checking against `locals()` to see if a variable was defined ala JavaScript's 'foo in window' syntax (but I suppose more portable), but I can't recall anything specifically caring about a global interpreter lock (instead of what the GIL has semantically provided, which is much different)
> What non-transparent GIL specific behavior are developers relying on exactly?
They are relying on behavior of a single environment. We similarly see a lot of issues moving threaded code to a new operating system in C/C++, because the underlying implementation details (cooperative threading, m:n hybrid threading, cpu-bound scheduling, I/O and signaling behavior) will hide bugs which came from mistakes and assumptions.
finally. you don't even have to read anything to work this out -- if things like dictionary access were no longer atomic that would imply that threaded code without locks could crash the interpreter, which isn't going to happen.
> GIL removal doesn't mean "make all of this lockless"
Literally speaking, that's exactly what "removal" means. As far as I can tell, GP was wondering why there's so much discussion about replacement, since simply removing the GIL wouldn't break single-threaded code.
I think the idea with localized access is that reads coming in after the initial commit can hit a nearby db instance, rather than the single global instance.
I agree with you about horizontal scalability in general though, the data isn't being partitioned. I guess read heavy loads can be scaled.
Even with a transaction, if the processing involves external side effects, e.g. sending and email, the rollback won't matter and you still get at least once.
There's no rollback, it's an atomic transaction. The certainty that messages are always handled completely or fail completely is one of the big design constraints that made the whole thing so hinky.
I dont have an answer, I'm just curious about something. I thought that the point of an outbox was that it was local, and could therefore be updated atomically along with any local db changes. What would happen if the orchestrator was unable to update the remote shared cache outbox after a local exception/rolled back transaction?
> What would happen if the orchestrator was unable to update the remote shared cache outbox after a local exception/rolled back transaction
To be sure, I understand your question, you are referring to the following scenario:
- Start global transaction
- Create shared cache entry
- Add events to the shared cache
- Local transaction fails
- Orchestrators starts rolling back local transaction
- Attempts to delete shared cache entry <-- THIS ONE FAILS
To ensure we don't have stale events in the cache, I would add a TTL, which is configurable for each job (such that you can have long-living events). So in case the orchestrator fails to delete the cache entry, they would be invalidated at some point by exceeding its TTL.
Yes, if the data is not already sorted. Thus it's O(n) for already sorted data and O(n log(n) + n) -- which simplifies to O(n log(n)) -- for arbitrary data.
