Installation Issue: '/usr/bin' exists but is not a virtual environment

[ralmond]

I followed these instructions:

# install the development version of packages, in case the
# issue is already fixed but not on CRAN yet.
install.packages("remotes")
remotes::install_github(sprintf("rstudio/%s", c("reticulate", "tensorflow", "keras")))
reticulate::miniconda_uninstall() # start with a blank slate
options(timeout=300) # Missing from your instructions, 
                                   # but necessary given my ISP
reticulate::install_miniconda()
keras::install_keras()

And got the following error mesages:

* Miniconda has been successfully installed at "~/.local/share/r-miniconda".
[1] "/home/ralmond/.local/share/r-miniconda"
> keras::install_keras()
Error in virtualenv_install(envname = envname, packages = packages, ignore_installed = pip_ignore_installed,  : 
  '/usr/bin' exists but is not a virtual environment

This is just running under Pop_OS (Ubuntu derivative) 20.10. the miniconda installation worked fine under Mac_OS.

Diagnostic info:

> py_config()
python:         /usr/bin/python3.9
libpython:      /usr/lib/python3.9/config-3.9-x86_64-linux-gnu/libpython3.9.so
pythonhome:     //usr://usr
version:        3.9.7 (default, Sep 10 2021, 14:59:43)  [GCC 11.2.0]
numpy:          /usr/lib/python3/dist-packages/numpy
numpy_version:  1.19.5
tensorflow:     [NOT FOUND]

NOTE: Python version was forced by RETICULATE_PYTHON
> tensorflow::tf_config()
Valid installation of TensorFlow not found.

Python environments searched for 'tensorflow' package:
 /usr/bin/python3.9

Python exception encountered:
 ModuleNotFoundError: No module named 'tensorflow'

You can install TensorFlow using the install_tensorflow() function.
 
> reticulate::import("tensorflow")
Error in py_module_import(module, convert = convert) : 
  ModuleNotFoundError: No module named 'tensorflow'
> reticulate::py_last_error()
$type
[1] "ModuleNotFoundError"

$value
[1] "No module named 'tensorflow'"

$traceback
[1] "  File \"/home/ralmond/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py\", line 39, in _import_hook\n    module = _import(\n"

$message
[1] "ModuleNotFoundError: No module named 'tensorflow'"

> sessionInfo()
R version 4.1.2 (2021-11-01)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Pop!_OS 21.10

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods  
[7] base     

other attached packages:
[1] reticulate_1.24

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.8       lattice_0.20-45  png_0.1-7       
 [4] zeallot_0.1.0    rappdirs_0.3.3   grid_4.1.2      
 [7] R6_2.5.1         jsonlite_1.7.3   magrittr_2.0.2  
[10] tfruns_1.5.0     cli_3.2.0        rlang_1.0.1     
[13] whisker_0.4      Matrix_1.4-0     generics_0.1.2  
[16] keras_2.8.0      tools_4.1.2      xfun_0.29       
[19] compiler_4.1.2   base64enc_0.1-3  tensorflow_2.8.0
[22] knitr_1.37   
>

[t-kalinowski]

NOTE: Python version was forced by RETICULATE_PYTHON

It looks like reticulate is not using the 'r-reticulate' miniconda installation at "~/.local/share/r-miniconda/envs/r-reticulate because you have the environment variable RETICULATE_PYTHON set, presumably to /usr/bin/python3.

If you unset it with Sys.unsetenv("RETICULATE_PYTHON"), does the installation succeed?

[ralmond]

1. Calling Sys.unsetenv("RETICULATE_PYTHON") doesn't change the output of py_config().
   I in fact set it using use_python("/usr/bin/python3") thinking that the problem (same error message) was that it wasn't finding the python named 'python3`

2. I think the correct command would be use_python("~/.local/share/r-miniconda/bin/python"), but after running that, I get:
   ``
   Restarting R session...

library(reticulate)
use_python("~/.local/share/r-miniconda/bin/python")
py_config()
python: /usr/bin/python3.9
libpython: /usr/lib/python3.9/config-3.9-x86_64-linux-gnu/libpython3.9.so
pythonhome: //usr://usr
version: 3.9.7 (default, Sep 10 2021, 14:59:43) [GCC 11.2.0]
numpy: /usr/lib/python3/dist-packages/numpy
numpy_version: 1.19.5

NOTE: Python version was forced by RETICULATE_PYTHON

keras::install_keras()
Error: .onLoad failed in loadNamespace() for 'tensorflow', details:
call: py_module_import(module, convert = convert)
error: ModuleNotFoundError: No module named 'tensorflow'`

[t-kalinowski]

Sorry you're having these troubles.

NOTE: Python version was forced by RETICULATE_PYTHON

As far as I know, this message only appears if the environment variable RETICULATE_PYTHON is set. Calling use_python() would give a message like.

NOTE: Python version was forced by use_python()

Perhaps the variable is being set in some unexpected place? (check .profile, .bash_profile, .bashrc, .Rprofile, ...).
Also, in the R session, what's the output of Sys.getenv("RETICULATE_PYTHON").

Another cause here might be that reticulate is trying to bind to the correct python, but then failing... that seems unlikely though, but you can try to diagnose by stepping through what reticulate does when you call reticulate::import("tensorflow") in a fresh R session.

[ralmond]

I'm not finding a reference to RETICULATE_PYTHON in .bashrc or .profile (and I don't have a .Rprofile). Not sure if this might be hidden in a system level file.

Sys.getenv("RETICULATE_PYTHON")
[1] "/usr/bin/python3.9"
> Sys.unsetenv("RETICULATE_PYTHON")
> Sys.getenv("RETICULATE_PYTHON")
[1] ""
> py_config()
python:         /usr/bin/python3.9
libpython:      /usr/lib/python3.9/config-3.9-x86_64-linux-gnu/libpython3.9.so
pythonhome:     //usr://usr
version:        3.9.7 (default, Sep 10 2021, 14:59:43)  [GCC 11.2.0]
numpy:          /usr/lib/python3/dist-packages/numpy
numpy_version:  1.19.5

NOTE: Python version was forced by RETICULATE_PYTHON

So I'm not managing to unset it.

The import is just telling me that tensorflow is not yet installed.

Restarting R session...

> library(reticulate)
> import("tensorflow")
Error in py_module_import(module, convert = convert) : 
  ModuleNotFoundError: No module named 'tensorflow'
> tensorflow::install_tensorflow()
Error: .onLoad failed in loadNamespace() for 'tensorflow', details:
  call: py_module_import(module, convert = convert)
  error: ModuleNotFoundError: No module named 'tensorflow'
> Sys.getenv("RETICULATE_PYTHON")
[1] "/usr/bin/python3.9"

So whatever is setting the RETICULATE_PYTHON variable is resetting it before a reboot.

[t-kalinowski]

Perhaps bash is not your default shell? Maybe check .zprofile or similar if you're using zsh? Also, .Renviron (both in ~ and your current directory). Also, maybe Renviron.site or Rprofile.site in your R installations "etc" directory.

If you don't end up find where RETICULATE_PYTHON is set, you can just call Sys.unsetenv() each R session before reticulate initializes python.

[ralmond]

I found where it was getting set.
I had set a project option in RStudio to point to python3 (probably because of an earlier python problem I was having).

When I unset this, it appears to fix the configuration problem:

Restarting R session...

> Sys.getenv("RETICULATE_PYTHON")
[1] ""
> library(reticulate)
> py_config()
python:         /home/ralmond/.local/share/r-miniconda/envs/r-reticulate/bin/python
libpython:      /home/ralmond/.local/share/r-miniconda/envs/r-reticulate/lib/libpython3.8.so
pythonhome:     /home/ralmond/.local/share/r-miniconda/envs/r-reticulate:/home/ralmond/.local/share/r-miniconda/envs/r-reticulate
version:        3.8.12 | packaged by conda-forge | (default, Jan 30 2022, 23:53:36)  [GCC 9.4.0]
numpy:          /home/ralmond/.local/share/r-miniconda/envs/r-reticulate/lib/python3.8/site-packages/numpy
numpy_version:  1.22.3

But I'm still getting problems on the installation:

> keras::install_keras()
Error: .onLoad failed in loadNamespace() for 'tensorflow', details:
  call: py_module_import(module, convert = convert)
  error: ModuleNotFoundError: No module named 'tensorflow'
> tensorflow::install_tensorflow()
Error: .onLoad failed in loadNamespace() for 'tensorflow', details:
  call: py_module_import(module, convert = convert)
  error: ModuleNotFoundError: No module named 'tensorflow'

I'm going to try re-executing the installation from github.
It seems to be working now, thanks for your help.

May I offer two suggestions for improving the docs:

1. add options(timeout=300) to the instructions before the install_miniconda() call, as others may need that as well.

2. Add a statement somewhere about checking the value of the RETICULATE_PYTHON environment if the installation fails

[t-kalinowski]

Thank you! We'll update the docs with the two suggestions.