Spring Dev Tools Support

[jamesward]

It'd be awesome if the shell (or other persistent runners: MCP, custom web API around an agent, etc) supported auto-reload with Spring Dev Tools.

When I tried this something got out-of-whack. (Note: I will try again and grab logs to share)

[jamesward]

As an example of this not working, see: https://github.com/jamesward/hello-embabel

This exposes an Embabel agent via Spring Web.

When dev tools is on the classpath, the agent is no longer found when using AgentInvocation.

[igordayen]

@jamesward
Could you please elaborate on "When dev tools is on the classpath, the agent is no longer found when using AgentInvocation."

upon restart agent is no longer operational? thank you

[jamesward]

Steps to reproduce:

1. Clone https://github.com/jamesward/hello-embabel
2. Uncomment the spring-boot-devtools dependency
3. Setup AWS credentials
4. Run: ./mvnw spring-boot:run
5. Make a GET request to: http://localhost:8080

This produces the error:

08:58:19.238 [tomcat-handler-0] ERROR [dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: java.lang.IllegalStateException: No agent with outputClass class embabelagent.JokeAndRating found.] with root cause
java.lang.IllegalStateException: No agent with outputClass class embabelagent.JokeAndRating found.

If you comment out the spring-boot-devtools dependency and re-start, everything works as expected.

[igordayen]

Thank you, James

[igordayen]

It'd be awesome if the shell (or other persistent runners: MCP, custom web API around an agent, etc) supported auto-reload with Spring Dev Tools.

When I tried this something got out-of-whack. (Note: I will try again and grab logs to share)

Experiment

Experimented with Shell. please refer to code:
https://github.com/igordayen/embabel-dev-research

Execution

mvn -Dmaven.test.skip=true spring-boot:run   -Dspring-boot.run.fork=true   

To force re-start:

• update spring.devtools.restart.enabled=true
• launch second instance of shell and "touch" application.properties

watch for:

16:16:35.639 [File Watcher] INFO  LocalDevToolsAutoConfiguration$RestartingClassPathChangeChangedEventListener - Restarting due to 1 class path change (1 addition, 0 deletions, 0 modifications)
16:16:35.640 [File Watcher] DEBUG LocalDevToolsAutoConfiguration$RestartingClassPathChangeChangedEventListener - Change set: [/home/igordayen/git/openai-agent-demo-with-embabel/src/main/resources [/home/igordayen/git/openai-agent-demo-with-embabel/src/main/resources/.application.properties.swp (ADD)]]

Summary

Combination of DevTools and Spring Shell works fine, as long as auto-reload gets disabled (see application.properties.

Sharing full problem statement in a separate window.

Would you please consider moving this to discussion if you feel that's is proper approach.

@jamesward please advise.

@johnsonr , @alexheifetz , @poutsma -FYI

[igordayen]

Spring Shell 3.x & Spring Boot DevTools Restart: Technical Analysis

Problem Statement

A Spring Boot application using Spring Shell 3.x enters an interactive CLI mode on startup. When Spring Boot DevTools is enabled with automatic restart, modifying any classpath resource triggers a DevTools restart cycle.

Observed behavior:

Before restart:

• Spring Shell loads normally

• Commands are available (e.g., OA → 29 completions)

After restart:

• Shell prompt remains visible

• Commands disappear (No command found for 'OA')

• TAB completion breaks

• Application becomes partially unresponsive

DevTools logs show restart triggered, but the application does not fully reinitialize

Manual termination (CTRL+C) is required to recover.

Impact: The interactive shell becomes unusable after any DevTools-triggered restart.

Root Cause: Why DevTools Restart Does Not Work

1 Spring Shell 3.x blocks the main thread

• Spring Shell 3.x starts an interactive terminal loop that:

• initializes JLine/Jansi terminal

• builds the command catalog

• enters a blocking read–eval–print loop

• never returns control to the main thread

2. DevTools restart requires the main thread to return

• DevTools restart works by:

• Stopping the old ApplicationContext

• Creating a new restart classloader

• Re-running the application entrypoint (main)

• Expecting the main thread to complete initialization again

But because Spring Shell 3.x never releases the main thread, DevTools cannot:

• reinitialize the shell

• rebuild the command catalog

• rebind the terminal

• reattach input/output streams

3. Classloader split occurs

After restart:

• The shell loop remains in the old classloader

• The commands reload in the new restart classloader

• The shell cannot see the new beans

• Command catalog becomes stale or empty

• This produces the exact failure:

No command found for 'OA'

4. Even full DevTools include/exclude tuning cannot fix this

Even when forcing DevTools to reload:

• Spring Shell core

• Spring Shell terminal subsystem

• JLine

• Jansi

• Project classes

• Custom agent JARs

…the shell loop still never reinitializes because DevTools cannot restart the application while the shell loop is blocking.

Conclusion: Spring Shell 3.x interactive mode is not compatible with DevTools restart.

5. Why Disabling Restart Fixes the Problem

Setting:

spring.devtools.restart.enabled=false

disables only the restart classloader, not DevTools itself.

This avoids:

• classloader splits

• stale shell loops

• broken command catalogs

• terminal reinitialization failures

The shell runs once, stays alive, and remains stable.

We still have benefits :

• enhanced logging

• classpath diagnostics

• LiveReload (if enabled)

• remote DevTools support

We lose only the restart feature — which Spring Shell 3.x cannot support anyway.

Options to Fix or Work Around the Limitation

Option A — Disable DevTools Restart (Recommended for Spring Shell 3.x)

spring.devtools.restart.enabled=false

Pros:

• Stable interactive shell

• No classloader issues

• No disappearing commands

• No terminal corruption

Cons:

• No hot reload of Java classes

Option B — Downgrade to Spring Shell 2.x

Spring Shell 2.x:

• does not block the main thread

• is restart-friendly

• reinitializes cleanly after context refresh

• supports built-in commands (help, quit, etc.)

Pros:

• Full DevTools restart compatibility

• Stable interactive shell

• No custom engineering required

Cons:

• Older API

• Different annotation model

• Less modern command framework

Option C — Redesign ShellCommand Beans to Make the Shell Restartable

This requires:

• running the shell loop in a separate thread

• detecting DevTools restarts (ContextClosedEvent)

• gracefully shutting down the old shell loop

• reinitializing terminal + JLine + Jansi

• rebuilding the command catalog

• starting a new shell loop bound to the new context

Pros:

• Full DevTools restart

• Full interactive shell

• Modern Spring Shell 3.x API

Cons:

• Custom engineering

• Must manage terminal lifecycle explicitly

• Must handle thread cleanup and rebind I/O

• Not supported by Spring Shell out of the box

Final Recommendation

For Spring Shell 3.x interactive applications, the correct and stable configuration is:

spring.devtools.restart.enabled=false

This avoids architectural conflicts between DevTools and the Spring Shell terminal loop, while preserving all other DevTools benefits.

If hot reload is a hard requirement, consider:

downgrading to Spring Shell 2.x, or

implementing a custom restartable shell loop

—but these require deliberate architectural changes.

[jamesward]

Thanks for the investigation! I think ultimately we want to:

1. Not break a non-Spring Shell app that has DevTools as a dependency
2. Support auto-reload with DevTools in both Spring Shell and non-Spring Shell apps

It sounds like we may need to start with some upstream fixes to DevTools & Spring Shell. Maybe we can get @philwebb to help.

Just to confirm, for the non-Spring Shell use cases, is the underlying issue something that has to be fixed upstream or is there something in Embabel that could be fixed?

[igordayen]

Thanks James, Option C - is more domain of Spring Shell /DevTools , solution should not be embabel-specific, IMO. we can have demo session if needed for spring team too.

[jamesward]

Issue filed with Spring Shell: spring-projects/spring-shell#1224

[jamesward]

Back to non-shell (web app use case), I'm not sure yet if that is an issue with DevTools or Embabel. Reproduction steps: #1133 (comment)

Can you all do more digging on that one as it seems like an issue with how Embabel does something with classloaders?

[philwebb]

With regards to "Spring Shell 3.x blocks the main thread", devtools should throw a SilentExitExceptionHandler in org.springframework.boot.devtools.restart.Restarter.immediateRestart() when the app first starts to trigger the first immediate reload. Perhaps we're not getting to that for some reason.

I suspect the issue is more likely to be with classloaders since that's historically been quite a cause of pain with devtools. If there is something we can improve on the Spring Boot side please open an issue at https://github.com/spring-projects/spring-boot/issues/new

[igordayen]

With regards to "Spring Shell 3.x blocks the main thread", devtools should throw a SilentExitExceptionHandler in org.springframework.boot.devtools.restart.Restarter.immediateRestart() when the app first starts to trigger the first immediate reload. Perhaps we're not getting to that for some reason

here is excerpt from log related to spring shell application:

starwars> 17:09:00.001 [File Watcher] INFO  LocalDevToolsAutoConfiguration$RestartingClassPathChangeChangedEventListener - Restarting due to 1 class path change (0 additions, 1 deletion, 0 modifications)
17:09:00.001 [File Watcher] DEBUG LocalDevToolsAutoConfiguration$RestartingClassPathChangeChangedEventListener - Change set: [/home/igordayen/git/openai-agent-demo-with-embabel/src/main/resources [/home/igordayen/git/openai-agent-demo-with-embabel/src/main/resources/.application.properties.swp (DELETE)]]
   ___  ___    _______ .___  ___. .______        ___      .______    _______  __         ___  ___
  /  / /  /   |   ____||   \/   | |   _  \      /   \     |   _  \  |   ____||  |        \  \ \  \
 /  / /  /    |  |__   |  \  /  | |  |_)  |    /  ^  \    |  |_)  | |  |__   |  |         \  \ \  \
<  < <  <     |   __|  |  |\/|  | |   _  <    /  /_\  \   |   _  <  |   __|  |  |          >  > >  >
 \  \ \  \    |  |____ |  |  |  | |  |_)  |  /  _____  \  |  |_)  | |  |____ |  `----.    /  / /  /
  \__\ \__\   |_______||__|  |__| |______/  /__/     \__\ |______/  |_______||_______|   /__/ /__/

Powered by Spring Boot 3.5.7
17:09:00.114 [restartedMain] INFO  ProjectNameApplication - Starting ProjectNameApplication using Java 21.0.5 with PID 2021365 (/home/igordayen/git/openai-agent-demo-with-embabel/target/class.......

[philwebb]

Could you please edit that and format the output in a ``` block? I can't read it.

[igordayen]

Could you please edit that and format the output in a ``` block? I can't read it.

done

[philwebb]

That [restartedMain] line means that the initial immediateRestart() method probably ran OK.

[igordayen]

thanks, i suspect that from DevTools perspective it's OK...but shell command catalog gets still broken....