Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Example Notebooks for 001259 #106

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

pauladkisson
Copy link
Contributor

@pauladkisson pauladkisson commented Dec 19, 2024

Streaming NWB files from 001259 currently does not work. Likely due to the same bug that makes no assets show up on the homepage: dandi/dandi-archive#2089

When I try to stream the nwbfile with this code:

from stream_nwbfile import stream_nwbfile
DANDISET_ID = '001259'
file_path = 'sub-m53/sub-m53_ses-231029_behavior+ecephys+image.nwb'
nwbfile, io = stream_nwbfile(DANDISET_ID, file_path)

I get this error:

---------------------------------------------------------------------------
ClientResponseError                       Traceback (most recent call last)
File /opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:423, in HTTPFileSystem._info(self, url, **kwargs)
    [421](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:421) try:
    [422](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:422)     info.update(
--> [423](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:423)         await _file_info(
    [424](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:424)             self.encode_url(url),
    [425](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:425)             size_policy=policy,
    [426](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:426)             session=session,
    [427](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:427)             **self.kwargs,
    [428](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:428)             **kwargs,
    [429](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:429)         )
    [430](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:430)     )
    [431](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:431)     if info.get("size") is not None:

File /opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:837, in _file_info(url, session, size_policy, **kwargs)
    [836](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:836) async with r:
--> [837](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:837)     r.raise_for_status()
    [839](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:839)     if "Content-Length" in r.headers:
    [840](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:840)         # Some servers may choose to ignore Accept-Encoding and return
    [841](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:841)         # compressed content, in which case the returned size is unreliable.

File /opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/aiohttp/client_reqrep.py:1161, in ClientResponse.raise_for_status(self)
   [1159](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/aiohttp/client_reqrep.py:1159)     self.release()
-> [1161](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/aiohttp/client_reqrep.py:1161) raise ClientResponseError(
   [1162](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/aiohttp/client_reqrep.py:1162)     self.request_info,
   [1163](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/aiohttp/client_reqrep.py:1163)     self.history,
   [1164](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/aiohttp/client_reqrep.py:1164)     status=self.status,
   [1165](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/aiohttp/client_reqrep.py:1165)     message=self.reason,
   [1166](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/aiohttp/client_reqrep.py:1166)     headers=self.headers,
   [1167](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/aiohttp/client_reqrep.py:1167) )

ClientResponseError: 403, message='Forbidden', url='https://dandiarchive.s3.amazonaws.com/blobs/9e8/772/9e87721c-c1da-4ed6-89b6-6f34e26f974b'

The above exception was the direct cause of the following exception:

FileNotFoundError                         Traceback (most recent call last)
Cell In[1], [line 4](vscode-notebook-cell:?execution_count=1&line=4)
      [2](vscode-notebook-cell:?execution_count=1&line=2) DANDISET_ID = '001259'
      [3](vscode-notebook-cell:?execution_count=1&line=3) file_path = 'sub-m53/sub-m53_ses-231029_behavior+ecephys+image.nwb'
----> [4](vscode-notebook-cell:?execution_count=1&line=4) nwbfile, io = stream_nwbfile(DANDISET_ID, file_path)

File ~/Documents/CatalystNeuro/DANDI/example-notebooks/001259/stream_nwbfile.py:32, in stream_nwbfile(DANDISET_ID, file_path)
     [30](https://file+.vscode-resource.vscode-cdn.net/Users/pauladkisson/Documents/CatalystNeuro/DANDI/example-notebooks/001259/~/Documents/CatalystNeuro/DANDI/example-notebooks/001259/stream_nwbfile.py:30)     s3_url = asset.get_content_url(follow_redirects=1, strip_query=True)
     [31](https://file+.vscode-resource.vscode-cdn.net/Users/pauladkisson/Documents/CatalystNeuro/DANDI/example-notebooks/001259/~/Documents/CatalystNeuro/DANDI/example-notebooks/001259/stream_nwbfile.py:31) fs = filesystem("http")
---> [32](https://file+.vscode-resource.vscode-cdn.net/Users/pauladkisson/Documents/CatalystNeuro/DANDI/example-notebooks/001259/~/Documents/CatalystNeuro/DANDI/example-notebooks/001259/stream_nwbfile.py:32) file_system = fs.open(s3_url, "rb")
     [33](https://file+.vscode-resource.vscode-cdn.net/Users/pauladkisson/Documents/CatalystNeuro/DANDI/example-notebooks/001259/~/Documents/CatalystNeuro/DANDI/example-notebooks/001259/stream_nwbfile.py:33) file = File(file_system, mode="r")
     [34](https://file+.vscode-resource.vscode-cdn.net/Users/pauladkisson/Documents/CatalystNeuro/DANDI/example-notebooks/001259/~/Documents/CatalystNeuro/DANDI/example-notebooks/001259/stream_nwbfile.py:34) io = NWBHDF5IO(file=file, load_namespaces=True)

File /opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/spec.py:1301, in AbstractFileSystem.open(self, path, mode, block_size, cache_options, compression, **kwargs)
   [1299](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/spec.py:1299) else:
   [1300](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/spec.py:1300)     ac = kwargs.pop("autocommit", not self._intrans)
-> [1301](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/spec.py:1301)     f = self._open(
   [1302](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/spec.py:1302)         path,
   [1303](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/spec.py:1303)         mode=mode,
   [1304](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/spec.py:1304)         block_size=block_size,
   [1305](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/spec.py:1305)         autocommit=ac,
   [1306](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/spec.py:1306)         cache_options=cache_options,
   [1307](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/spec.py:1307)         **kwargs,
   [1308](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/spec.py:1308)     )
   [1309](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/spec.py:1309)     if compression is not None:
   [1310](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/spec.py:1310)         from fsspec.compression import compr

File /opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:362, in HTTPFileSystem._open(self, path, mode, block_size, autocommit, cache_type, cache_options, size, **kwargs)
    [360](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:360) kw.update(kwargs)
    [361](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:361) info = {}
--> [362](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:362) size = size or info.update(self.info(path, **kwargs)) or info["size"]
    [363](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:363) session = sync(self.loop, self.set_session)
    [364](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:364) if block_size and size and info.get("partial", True):

File /opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/asyn.py:118, in sync_wrapper.<locals>.wrapper(*args, **kwargs)
    [115](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/asyn.py:115) @functools.wraps(func)
    [116](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/asyn.py:116) def wrapper(*args, **kwargs):
    [117](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/asyn.py:117)     self = obj or args[0]
--> [118](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/asyn.py:118)     return sync(self.loop, func, *args, **kwargs)

File /opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/asyn.py:103, in sync(loop, func, timeout, *args, **kwargs)
    [101](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/asyn.py:101)     raise FSTimeoutError from return_result
    [102](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/asyn.py:102) elif isinstance(return_result, BaseException):
--> [103](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/asyn.py:103)     raise return_result
    [104](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/asyn.py:104) else:
    [105](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/asyn.py:105)     return return_result

File /opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/asyn.py:56, in _runner(event, coro, result, timeout)
     [54](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/asyn.py:54)     coro = asyncio.wait_for(coro, timeout=timeout)
     [55](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/asyn.py:55) try:
---> [56](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/asyn.py:56)     result[0] = await coro
     [57](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/asyn.py:57) except Exception as ex:
     [58](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/asyn.py:58)     result[0] = ex

File /opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:436, in HTTPFileSystem._info(self, url, **kwargs)
    [433](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:433)     except Exception as exc:
    [434](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:434)         if policy == "get":
    [435](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:435)             # If get failed, then raise a FileNotFoundError
--> [436](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:436)             raise FileNotFoundError(url) from exc
    [437](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:437)         logger.debug("", exc_info=exc)
    [439](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/schneider_notebook_env/lib/python3.12/site-packages/fsspec/implementations/http.py:439) return {"name": url, "size": None, **info, "type": "file"}

FileNotFoundError: https://dandiarchive.s3.amazonaws.com/blobs/9e8/772/9e87721c-c1da-4ed6-89b6-6f34e26f974b

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant