Reload code location not working

[kennycontreras]

Hello community 👋

I'd like some insights into what could be wrong with my local Dagster setup. When I try to reload the code location, it doesn't work, even though I can see in the logs that the modules are being loaded again.

I have a local docker image running Dagster.

Before I was using the @repository decorator, the reload always worked fine. Recently, I migrated the code to use Definitions, and the issue started.

Here are some of my files:

Dockerfile

FROM base_dagster_image

RUN mkdir -p /opt/dagster/app
WORKDIR /opt/dagster/app

COPY . .
RUN pip install -r requirements.txt

EXPOSE 4000

CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-w", "workspace.yaml"]
#CMD ["dagster", "code-server", "start", "-h", "0.0.0.0", "-p", "4000", "-w", "workspace.yaml"]

I have tried with code-server and grpc

workspace.yaml

load_from:
  - python_file:
      relative_path: "definitions.py"
      attribute: "batch"
      location_name: "batch"

docker-compose.yml

version: '3.4'
services:
  dagster:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "3000:3000"
    command: >
      bash -c "dagster-webserver -h 0.0.0.0 -w /opt/dagster/app/workspace.yaml"
    working_dir: /opt/dagster/app/
    volumes:
      - ./:/opt/dagster/app
    environment:
      - DAGSTER_HOME=/tmp

Based on the logs, I know that my definitions.py is reloading the modules, but even then, the actual code is not the one with the latest changes. For example, If one pipeline has an error that I fixed. I click reload, but the error is still there.

The reloading string there is coming from the definitions.py implementation.

definitions.py

from dagster import Definitions, AssetsDefinition, ScheduleDefinition, SensorDefinition, JobDefinition
from dagster._core.definitions.partitioned_schedule import UnresolvedPartitionedAssetScheduleDefinition
from common.utils import recursively_find_instances

dagster_assets = recursively_find_instances(root="batch", types=(AssetsDefinition,))
dagster_schedules = recursively_find_instances(root="batch", types=(ScheduleDefinition,
                                                                    UnresolvedPartitionedAssetScheduleDefinition))
dagster_sensors = recursively_find_instances(root="batch", types=(SensorDefinition,))
dagster_jobs = recursively_find_instances(root="batch", types=(JobDefinition,))

batch = Definitions(
    assets=dagster_assets,
    schedules=dagster_schedules,
    sensors=dagster_sensors,
    jobs=dagster_jobs
    # resources={...}
)

Update:

To provide more context, as I continue debugging what may be the issue. The hot reloading for the container is working.
If I make some local changes, they are also changed in the container. I double-checked this by getting inside the container and doing a simple cat to the files. Even then, Dagster doesn't acknowledge the changes with the reload button. Could it be some catch when the importlib for modules happens?

Any insights or recommendations would be really appreciated.

Thanks!

[kennycontreras]

Edit: Fixed the problem with importlib.reload(module_name) and it's working perfectly.

[C-Loftus]

@kennycontreras Could you clarify where you put that line importlib.reload(module_name)? I am also having issues with this and am wondering if I have the same issue.

i.e. if I have code that throws an error on it being loaded, I can remove that reload the module and the error won't be there, but at runtime it still appears; the new code appears to not be reloaded in some parts of the dagster runtime at least

Also see that you have the following commented out. Any reason for that? I thought you need for reloading: #14709

#CMD ["dagster", "code-server", "start", "-h", "0.0.0.0", "-p", "4000", "-w", "workspace.yaml"]