Redis Dispatcher V2

[joshStillerman]

I have been thinking about the Redis Dispatcher

Here are some random ideas. Please feel free to shoot them down!

• It exposes some severe problems with threads and mdip based IO (not part of this discussion)
• The basic idea(s) are great
• Do we need authentication or some 'shared secret'
  • can we get this for free from Redis ?
• The servers need to be able to do commands like
  • show server
    • show tasks if servers can do more than one thing
  • abort task(s)
  • abort phase
  • restart
  • stop
• The servers could be spun up for each task ? Inetd like ?
• We need good logging
• Interdependent shell scripts
• Jobs that are started externally and just have completions (d-tacq auto store for example)
• Jobs that are long running
• Jobs can extend past end of shot
• Need to know what shots are active
  • Stop/deactivate them
  • show experiment-name - output active shots
  • show experiment-name shot - status of 'shot'
  • ? access to logs this way ? probably not
• The keys for a shot should be populated by whatever process does the build_tables
• Straw man for the keys
  • Experiment:shot:phase-name
  • Experiment:shot:phase-name:server-name
  • Experiment:shot:phase-name:server-name:pathname
  • Experiment:shot:phase-name:server-name:pathname:schedule (n or boolean)
  • Experiment:shot:phase-name:server-name:pathname:job-status
    • not dispatched
    • running (more detail here for async jobs?)
    • done
    • do not dispatch (off)
    • will not dispatch - dependency can not be met
  • Experiment:shot:phase-name:server-name:pathname:status
    • pending
    • success
    • canceled
    • stopped
  • Experiment:shot:phase-name:server-name:pathname:serverid (id of server running or has run)
    • how do we ID servers? host + pid? something else?
• we can store date/times in some of the redis keys?
• We can use redis lua scripts to atomically claim jobs:
  • https://lucaspin.medium.com/atomicity-in-redis-operations-a1d7bc9f4a90
• Basic Operation
  • plant state interface / shot-phase poker
    • create pulse
    • build table
      • remove previous keys for this experiment / shot
      • find all actions
      • make the keys
      • notify displayers and action servers(if necessary) of new shot
    • do phase
      • notify servers to do the phase
  • action server
    • ? respond to 'check-in' messages ?
    • show job(s)
    • abort job(s)
    • Interactions with systemd or ...
      • stop
      • restart
      • start?
    • do phase
      • while I can find and claim job that is mine (this lua script could deal with dependences too)
        • do it or start it
        • update status

[GabrieleManduchi]

Most of them are good points deserving further discussion. Maybe time to organize the remote MDSplus meeting?  (as pointed out in my mail earlier)                 Ciao                 Gabriele

…

On 3/29/2023 3:33 PM, Josh Stillerman wrote: I have been thinking about the Redis Dispatcher Here are some random ideas. Please feel free to shoot them down! * It exposes some severe problems with threads and mdip based IO (not part of this discussion) * The basic idea(s) are great * The servers need to be able to do commands like o show server + show tasks if servers can do more than one thing o abort task(s) o abort phase o restart o stop * The servers could be spun up for each task ? Inetd like ? * We need good logging * Interdependent shell scripts * Jobs that are started externally and just have completions (d-tacq auto store for example) * Jobs that are long running * Jobs can extend past end of shot * Need to know what shots are active o Stop/deactivate them o show experiment-name - output active shots o show experiment-name shot - status of 'shot' o ? access to logs this way ? probably not * The keys for a shot should be populated by whatever process does the build_tables * Straw man for the keys o Experiment:shot:phase-name o Experiment:shot:phase-name:server-name o Experiment:shot:phase-name:server-name:pathname o Experiment:shot:phase-name:server-name:pathname:schedule (n or boolean) o Experiment:shot:phase-name:server-name:pathname:job-status + not dispatched + running (more detail here for async jobs?) + done + do not dispatch (off) + will not dispatch - dependency can not be met o Experiment:shot:phase-name:server-name:pathname:status + pending + success + canceled + stopped o Experiment:shot:phase-name:server-name:pathname:serverid (id of server running or has run) + how do we ID servers? host + pid? something else? * we can store date/times in some of the redis keys? * We can use redis lua scripts to atomically claim jobs: o https://lucaspin.medium.com/atomicity-in-redis-operations-a1d7bc9f4a90 * Basic Operation o plant state interface / shot-phase poker + create pulse + build table # remove previous keys for this experiment / shot # find all actions # make the keys # notify displayers and action servers(if necessary) of new shot + do phase # notify servers to do the phase o action server + ? respond to 'check-in' messages ? + show job(s) + abort job(s) + Interactions with systemd or ... # stop # restart # start? + do phase # while I can find and claim job that is mine (this lua script could deal with dependences too) * do it or start it * update status — Reply to this email directly, view it on GitHub <#2552>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCLESC6JLQ3JOAGPSRPTZDW6Q24JANCNFSM6AAAAAAWL7SMQI>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- Gabriele Manduchi Istituto Gas Ionizzati del CNR Consorzio RFX - Associazione EURATOM/ENEA sulla Fusione Corso Stati Uniti 4, 35127 Padova - Italy ph +39-049-829-5039/-5000 fax +39-049-8700718 ***@***.***, http://www.igi.cnr.it

[joshStillerman]

The monitor can be a pure javascript application
the poker / state thing should probably be python

[joshStillerman]

should the dispatcher (the thing that does build tables, do phase, etc.) be a service or just a program that users can run?