Moar threadsafe moar better #101

JamesWrigley · 2024-05-23T15:54:55Z

This is a rebased version of #4, it should be ready to merge. Fixes #73.

(made after discussing with @jpsamaroo)

JamesWrigley · 2024-05-24T12:25:09Z

I tracked the test failures down to: JuliaLang/julia#53326
The workers are loading the builtin version of Distributed from the sysimg while the master is using the development version, and because we changed the definition of the Worker struct in this PR we get a serialization error when doing a remotecall to another worker because the master and worker are using different types.

I think this should be fixed in Base, made a PR here: JuliaLang/julia#54571
That should be merged before this one.

codecov · 2024-05-25T11:06:38Z

Codecov Report

Attention: Patch coverage is 86.56716% with 9 lines in your changes missing coverage. Please review.

Project coverage is 79.30%. Comparing base (6a0383b) to head (abafb79).
Report is 1 commits behind head on master.

Files	Patch %	Lines
src/cluster.jl	89.09%	6 Missing ⚠️
src/managers.jl	50.00%	1 Missing ⚠️
src/messages.jl	0.00%	1 Missing ⚠️
src/process_messages.jl	85.71%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #101      +/-   ##
==========================================
+ Coverage   79.18%   79.30%   +0.11%     
==========================================
  Files          10       10              
  Lines        1898     1918      +20     
==========================================
+ Hits         1503     1521      +18     
- Misses        395      397       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

JamesWrigley · 2024-05-25T11:17:06Z

Alrighty, after a force push to trigger CI with latest nightly we are back in the green 🥳 I think this is ready to be merged now.

jonas-schulze

I can confirm that the original code snippet from #73 (comment) is working now. 🚀

Will this be compatible to or backported, if necessary, to the next Julia LTS?

jonas-schulze · 2024-05-30T13:05:56Z

src/cluster.jl

    @async manage(w.manager, w.id, w.config, :register)
    # wait for rr_ntfy_join with timeout
    timedout = false
-    @async (sleep($timeout); timedout = true; put!(rr_ntfy_join, 1))
+    @async begin
+        sleep($timeout)
+        timedout = true
+        put!(rr_ntfy_join, 1)
+    end


Do these tasks need an errormonitor?

Yeah I think that makes sense, added in 03d7384.

jonas-schulze · 2024-05-30T13:08:08Z

test/threads.jl

+    ws = ts = product(1:2, 1:2)
+    @testset "from worker $w1 to $w2 via 1" for (w1, w2) in ws
+        @testset "from thread $w1.$t1 to $w2.$t2" for (t1, t2) in ts
+          # We want (the default) lazyness, so that we wait for `Worker.c_state`!


Suggested change

# We want (the default) lazyness, so that we wait for `Worker.c_state`!

# We want (the default) laziness, so that we wait for `Worker.c_state`!

Fixed in f4576aa.

jonas-schulze · 2024-05-30T13:12:29Z

test/threads.jl

+            end
+
+            # Wait on the spawned tasks on the owner
+            @sync begin


IMO, this sync point should fail fast, if necessary:

Suggested change

@sync begin

Base.Experimental.@sync begin

See JuliaLang/julia#42239 (comment)

Sure that makes sense, I refactored the code to use timedwait() in f4576aa.

JamesWrigley · 2024-06-02T17:58:02Z

Will this be compatible to or backported, if necessary, to the next Julia LTS?

I'll leave that for someone more qualified to properly answer, but FWIW if 1.11 is chosen as the next LTS then it'll be possible to upgrade Distributed now that it's an excised stdlib 🐙

JamesWrigley · 2024-06-03T09:21:26Z

One other thing I noticed is that this should probably be using Threads.@spawn everywhere instead of @async, but I'll leave that for another PR.

jonas-schulze

Are there any updates on this?

JamesWrigley · 2024-06-24T14:09:39Z

I believe @JBlaschke was going to have a look at it. In the meantime I see the branch is out of date, so I'll rebase it.

JBlaschke · 2024-06-24T15:09:02Z

@JamesWrigley @jonas-schulze I'll be working on this this week. Just getting back up to speed after long travel...

zsz00 · 2024-07-21T03:39:20Z

Are there any updates on this? @JamesWrigley @JBlaschke

JamesWrigley · 2024-07-21T09:34:36Z

Not from me, still need someone to review it. I believe the hesitation to merge comes from Distributed being used to run the Julia tests, so it's quite critical that this works properly.

But in the meantime you can ] dev https://github.com/JamesWrigley/Distributed.jl.git#jps/threadsafe_workerstate on 1.11 to use this branch.

gbaraldi · 2024-07-23T18:37:41Z

src/cluster.jl

-    c_state::Condition      # wait for state changes
-    ct_time::Float64        # creation time
-    conn_func::Any          # used to setup connections lazily
+    @atomic state::WorkerState


if state is always read/written from inside a lock this doesn't need to be atomic as the lock should have the correct barriers.

I don't think that's guaranteed? From a cursory grep through cluster.jl I see plenty of reads outside of a lock.

gbaraldi · 2024-07-23T18:39:23Z

Since we are making things threadsafe I would look at all @async uses because we want to deprecate it.

JamesWrigley · 2024-07-23T19:28:32Z

Ok, I replaced all uses of @async with Threads.@spawn in 7cac33f. But there are still a couple places where Base.@async_unwrap is used and I can't find an easy analogue for Threads.@spawn, I'll see if I can write a replacement.

JamesWrigley · 2024-07-23T20:33:12Z

After thinking about it for a bit, I can't come up with a decent replacement short of basically reimplementing Threads.@spawn. We could replace @async_unwrap with Threads.@spawn, but it would change the type of any exceptions thrown from whatever they currently are to a TaskFailedException, which is technically breaking.

I'd suggest keeping it in for now, we can add support for unwrapping exceptions to Threads.@spawn and switch in a future release.

jonas-schulze · 2024-07-25T08:20:05Z

test/threads.jl

+            # Wait on the spawned tasks on the owner. Note that we use
+            # timedwait() instead of @sync to avoid deadlocks.
+            t1 = Threads.@spawn fetch_from_owner(wait, recv)
+            t2 = Threads.@spawn fetch_from_owner(wait, send)
+            @test timedwait(() -> istaskdone(t1), 5) == :ok
+            @test timedwait(() -> istaskdone(t2), 5) == :ok


Sorry to chime in so late after #101 (comment); I noticed because GitHub unfolded all my previous comments.

I like the timedwait, which is what I used in JuliaLang/julia#37905. However, the timedwait has been the main reason (I think) why my first PR was reverted (JuliaLang/julia#38112). The second attempt (https://github.com/JuliaLang/julia/pull/38134/files) didn't use timedwait. I remain in favor of timedwait but wanted to refresh the information, as it has been a while.

I see, thanks. My view is that if this fails we're going to end up with a timeout somewhere no matter what, either in CI or the tests themselves. And my preference would be to have the timeout in the tests so we can have some control over it. I bumped it to 60s in abafb79 but I'm happy to increase that if people think it's too low. @vchuravy, @vtjnash, does that sound ok?

JamesWrigley · 2024-08-28T08:27:12Z

Wee progress update for those following this, after some discussion with @vchuravy on Slack I decided to:

Try to run the Julia tests in the Distributed.jl CI since that's what was failing last time.
Attempt to audit the code for correctness.

Which I'll get to at... some point.

JamesWrigley mentioned this pull request May 23, 2024

Thread safety improvements #4

Closed

JamesWrigley force-pushed the jps/threadsafe_workerstate branch from 9a8469d to ec8bce0 Compare May 25, 2024 10:57

jonas-schulze reviewed May 30, 2024

View reviewed changes

jonas-schulze approved these changes Jun 24, 2024

View reviewed changes

JamesWrigley force-pushed the jps/threadsafe_workerstate branch 2 times, most recently from e3205d8 to fa9d645 Compare June 24, 2024 14:11

JamesWrigley force-pushed the jps/threadsafe_workerstate branch from fa9d645 to b140754 Compare July 21, 2024 09:36

gbaraldi reviewed Jul 23, 2024

View reviewed changes

jpsamaroo and others added 6 commits July 25, 2024 09:51

Add gitignore

2923ac2

Make worker state variable threadsafe

f621a8e

use atomic field

1a0d6fb

init_multi: Be more thread-safe

342302f

Use errormonitor() in a few places

0377cda

Replace @async with Threads.@Spawn

01c33e9

JamesWrigley force-pushed the jps/threadsafe_workerstate branch from 7cac33f to 01c33e9 Compare July 25, 2024 07:51

jonas-schulze reviewed Jul 25, 2024

View reviewed changes

fixup! Make worker state variable threadsafe

abafb79

JamesWrigley mentioned this pull request Oct 29, 2024

Such threadsafe, much wow JuliaParallel/DistributedNext.jl#4

Merged

m-fila mentioned this pull request Oct 30, 2024

use fork of distributed with fix for thread-safety key4hep/key4hep-julia-fwk#61

Closed

jpsamaroo mentioned this pull request Dec 2, 2024

Start julia with multiple threads by default JuliaLang/julia#43672

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Moar threadsafe moar better #101

Moar threadsafe moar better #101

JamesWrigley commented May 23, 2024

JamesWrigley commented May 24, 2024

codecov bot commented May 25, 2024 •

edited

Loading

JamesWrigley commented May 25, 2024

jonas-schulze left a comment

jonas-schulze May 30, 2024

JamesWrigley Jun 2, 2024

jonas-schulze May 30, 2024

JamesWrigley Jun 2, 2024

jonas-schulze May 30, 2024

JamesWrigley Jun 2, 2024

JamesWrigley commented Jun 2, 2024

JamesWrigley commented Jun 3, 2024

jonas-schulze left a comment

JamesWrigley commented Jun 24, 2024

JBlaschke commented Jun 24, 2024

zsz00 commented Jul 21, 2024

JamesWrigley commented Jul 21, 2024 •

edited

Loading

gbaraldi Jul 23, 2024 •

edited

Loading

JamesWrigley Jul 23, 2024

gbaraldi commented Jul 23, 2024

JamesWrigley commented Jul 23, 2024

JamesWrigley commented Jul 23, 2024

jonas-schulze Jul 25, 2024

JamesWrigley Jul 25, 2024 •

edited

Loading

JamesWrigley commented Aug 28, 2024

	# We want (the default) lazyness, so that we wait for `Worker.c_state`!
	# We want (the default) laziness, so that we wait for `Worker.c_state`!

Moar threadsafe moar better #101

Are you sure you want to change the base?

Moar threadsafe moar better #101

Conversation

JamesWrigley commented May 23, 2024

JamesWrigley commented May 24, 2024

codecov bot commented May 25, 2024 • edited Loading

Codecov Report

JamesWrigley commented May 25, 2024

jonas-schulze left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JamesWrigley commented Jun 2, 2024

JamesWrigley commented Jun 3, 2024

jonas-schulze left a comment

Choose a reason for hiding this comment

JamesWrigley commented Jun 24, 2024

JBlaschke commented Jun 24, 2024

zsz00 commented Jul 21, 2024

JamesWrigley commented Jul 21, 2024 • edited Loading

gbaraldi Jul 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gbaraldi commented Jul 23, 2024

JamesWrigley commented Jul 23, 2024

JamesWrigley commented Jul 23, 2024

Choose a reason for hiding this comment

JamesWrigley Jul 25, 2024 • edited Loading

Choose a reason for hiding this comment

JamesWrigley commented Aug 28, 2024

codecov bot commented May 25, 2024 •

edited

Loading

JamesWrigley commented Jul 21, 2024 •

edited

Loading

gbaraldi Jul 23, 2024 •

edited

Loading

JamesWrigley Jul 25, 2024 •

edited

Loading