URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI even through PYICEBERG_HOME is set to $HOME

[lk-1984]

Apache Iceberg version

None

Please describe the bug 🐞

I have set PYICEBERG_HOME="$(pwd)" as told in the docs.

https://py.iceberg.apache.org/api/

This information must be placed inside a file called .pyiceberg.yaml located either in the $HOME or %USERPROFILE% directory (depending on whether the operating system is Unix-based or Windows-based, respectively) or in the $PYICEBERG_HOME directory (if the corresponding environment variable is set).

MINIO_REGION=us-east-1
PWD=/home/pyiceberg
MINIO_PORT=9100
HOME=/home/pyiceberg
HIVE_PORT=9083
VIRTUAL_ENV=/home/pyiceberg/.venv
TERM=xterm
MINIO_USERNAME=admin
USER=pyiceberg
HIVE_HOSTNAME=localhost
SHLVL=1
PYICEBERG_HOME=/home/pyiceberg
MINIO_HOSTNAME=localhost
VIRTUAL_ENV_PROMPT=(pyiceberg)
PS1=(pyiceberg)
MINIO_PASSWORD=adminadmin
PATH=/home/pyiceberg/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/pyiceberg/.local/bin
_=/usr/bin/printenv
catalog:
  hive:
    uri: thrift://localhost:9083
    s3.endpoint: http://localhost:9100
    s3.access-key-id: admin
    s3.secret-access-key: adminadmin
    s3.region: us-east-1
total 44
drwx------ 1 pyiceberg pyiceberg 4096 Mar  6 16:48 .
drwxr-xr-x 1 root      root      4096 Mar  6 16:00 ..
-rw-r--r-- 1 pyiceberg pyiceberg   18 Apr 30  2024 .bash_logout
-rw-r--r-- 1 pyiceberg pyiceberg  141 Apr 30  2024 .bash_profile
-rw-r--r-- 1 pyiceberg pyiceberg  594 Mar  6 16:35 .bashrc
drwxr-xr-x 1 pyiceberg pyiceberg 4096 Mar  6 16:35 .cache
drwxr-xr-x 1 pyiceberg pyiceberg 4096 Mar  6 16:35 .local
-rw-r--r-- 1 pyiceberg pyiceberg  179 Mar  6 16:48 .pyiceberg.yaml
drwxr-xr-x 4 pyiceberg pyiceberg 4096 Mar  6 16:35 .venv
-rwxr-xr-x 1 pyiceberg pyiceberg  446 Mar  6 16:34 entrypoint.sh
URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI

I have Pyiceberg 0.9.0, and Python 3.12.

If I just leave PYICEBERG_HOME unset, the pyiceberg CLI still does not find .pyiceberg.yaml.

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[kevinjqliu]

Just to clarify

1. You set PYICEBERG_HOME to /home/pyiceberg in the environment variables
   I see above

PYICEBERG_HOME=/home/pyiceberg

2. /home/pyiceberg/.pyiceberg.yaml exists and contains the following

catalog:
  hive:
    uri: thrift://localhost:9083
    s3.endpoint: http://localhost:9100
    s3.access-key-id: admin
    s3.secret-access-key: adminadmin
    s3.region: us-east-1

What command did you run to see the error

URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI

[lk-1984]

The command was "list".

Btw, I pulled your repo, and built the thing in venv that has Python 3.11..

(.venv) dev@foobar iceberg-python % pyiceberg --help
Usage: pyiceberg [OPTIONS] COMMAND [ARGS]...

Options:
  --catalog TEXT
  --verbose BOOLEAN
  --output [text|json]
  --ugi TEXT
  --uri TEXT
  --credential TEXT
  --help                Show this message and exit.

Commands:
  create      Operation to create a namespace.
  describe    Describe a namespace or a table.
  drop        Operations to drop a namespace or table.
  files       List all the files of the table.
  list        List tables or namespaces.
  list-refs   List all the refs in the provided table.
  location    Return the location of the table.
  properties  Properties on tables/namespaces.
  rename      Rename a table.
  schema      Get the schema of the table.
  spec        Return the partition spec of the table.
  uuid        Return the UUID of the table.
  version     Print pyiceberg version.
(.venv) dev@foobar iceberg-python % touch .pyiceberg.yaml
(.venv) dev@foobar iceberg-python % nvim .
(.venv) dev@foobariceberg-python % pyiceberg list
URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI
(.venv) dev@foobar iceberg-python % pwd
/Users/dev/Projects/external/iceberg-python
(.venv) dev@foobar iceberg-python % export HOME=$(pwd)
(.venv) dev@foobar ~ % echo $HOME
/Users/dev/Projects/external/iceberg-python
(.venv) dev@foobar ~ % pyiceberg list
URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI
(.venv) dev@foobar~ % export PYICEBERG_HOME=$(pwd)
(.venv) dev@foobar~ % echo $PYICEBERG_HOME
/Users/dev/Projects/external/iceberg-python
(.venv) dev@foobar ~ % pyiceberg list
URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI
(.venv) dev@foobar~ % cat $HOME/.pyiceberg.yaml
catalog:
  hive:
    uri: thrift://localhost:9083
    s3.endpoint: http://localhost:9100
    s3.access-key-id: admin
    s3.secret-access-key: adminadmin
    s3.region: us-east-1
(.venv) dev@foobar~ % cat $PYICEBERG_HOME/.pyiceberg.yaml
catalog:
  hive:
    uri: thrift://localhost:9083
    s3.endpoint: http://localhost:9100
    s3.access-key-id: admin
    s3.secret-access-key: adminadmin
    s3.region: us-east-1

[kevinjqliu]

could you try pyiceberg list --catalog hive?

[lk-1984]

Same:

(.venv) dev@foobar~ % pyiceberg list --catalog hive
URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI

[lk-1984]

With some debug print:

(.venv) dev@foobar ~ % pyiceberg list --catalog hive
/Users/dev/Projects/external/iceberg-python
{'catalog': {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}}
URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI

[lk-1984]

Ok, there is new_config with uri:

(.venv) dev@foobar~ % pyiceberg list --catalog hive
/Users/dev/Projects/external/iceberg-python
{'catalog': {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}}
{'catalog': {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}}
new_config: {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}
new_config: {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}
new_config: {'catalog': {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}, 'home': '/Users/dev/Projects/external/iceberg-python'}
new_config: {}
URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI
(.venv) dev@foobar~ %

[lk-1984]

https://github.com/apache/iceberg-python/blob/main/pyiceberg/utils/config.py#L152

This is is the reason why an empty config is returned, because the name of the catalog is "default", instead of "hive", as it supposed to be.

    def get_catalog_config(self, catalog_name: str) -> Optional[RecursiveDict]:
        print(f"self.config: {self.config}")
        print(f"CATALOG: {CATALOG}")
        if CATALOG in self.config:
            catalog_name_lower = catalog_name.lower()
            catalogs = self.config[CATALOG]
            print(f"catalog_name_lower: {catalog_name_lower}")
            print(f"catalogs: {catalogs}")
            print(f"not isinstance(catalogs, dict): {not isinstance(catalogs, dict)}")
            if not isinstance(catalogs, dict):
                raise ValueError(f"Catalog configurations needs to be an object: {catalog_name}")
            print(f"catalog_name_lower in catalogs: {catalog_name_lower in catalogs:}")
            if catalog_name_lower in catalogs:
                catalog_conf = catalogs[catalog_name_lower]
                assert isinstance(catalog_conf, dict), f"Configuration path catalogs.{catalog_name_lower} needs to be an object"
                return catalog_conf
        print("THIS SHOULD NOT HAPPEN, BUT IT DID")
        return None

self.config: {'catalog': {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}, 'home': '/Users/dev/Projects/external/iceberg-python'}
CATALOG: catalog
catalog_name_lower: default
catalogs: {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}
not isinstance(catalogs, dict): False
catalog_name_lower in catalogs: False
THIS SHOULD NOT HAPPEN, BUT IT DID
env: None
new_config: {}
env: {}
catalog_properties: {}
URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI

Also pyiceberg list --catalog hive does not work, because the --catalog is not parsed and it is None.

If the --catalog is not given, then it will try to use default catalog that does not exists. If the --catalog flag is given, it is not parsed, and therefore it is the same result.

@click.group()
@click.option("--catalog")
@click.option("--verbose", type=click.BOOL)
@click.option("--output", type=click.Choice(["text", "json"]), default="text")
@click.option("--ugi")
@click.option("--uri")
@click.option("--credential")
@click.pass_context
def run(
    ctx: Context,
    catalog: Optional[str],
    verbose: bool,
    output: str,
    ugi: Optional[str],
    uri: Optional[str],
    credential: Optional[str],
) -> None:
    print(f"--catalog: {catalog}")
    print(f"--uri: {uri}")

(.venv) dev@foobar~ % pyiceberg list --catalog hive --uri foobar
/Users/dev/Projects/external/iceberg-python
{'catalog': {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}}
{'catalog': {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}}
new_config: {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}
new_config: {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}
new_config: {'catalog': {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}, 'home': '/Users/dev/Projects/external/iceberg-python'}
_ENV_CONFIG: <pyiceberg.utils.config.Config object at 0x120c461d0>
_ENV_CONFIG.config: {'catalog': {'hive': {'uri': 'thrift://localhost:9083', 's3.endpoint': 'http://localhost:9100', 's3.access-key-id': 'admin', 's3.secret-access-key': 'adminadmin', 's3.region': 'us-east-1'}}, 'home': '/Users/dev/Projects/external/iceberg-python'}
--catalog: None
--uri: None

[lk-1984]

So, if the user of the pyiceberg CLI would be able to give the catalog name hive, then it would work like when using the pyiceberg as a library - load_catalog(name="hive").

Now, in reality, the pyiceberg CLI does expect that the user would give the name of the catalog through the --catalog flag as there is no logic whatsoever for identifying which catalog in the .pyiceberg.yaml file would be selected after parsing the configuration.

So, if I hard code catalog_name_lower = "hive" at the line 151, then it works. To achieve the same through configuration file, you must provide logic for selecting the hive from the configuration file, or force user to use the --catalog flag, and make sure that the --catalog flag is parsed.

https://github.com/apache/iceberg-python/blob/main/pyiceberg/utils/config.py#L151

[lk-1984]

There is actually one combination that works, and which is documented in the click help.

Usage: pyiceberg [OPTIONS] COMMAND [ARGS]...

pyiceberg --catalog hive list

This indeed works.

However, the user should be instructed correctly when they do it wrong, like pyiceberg list --catalog hive..

URI missing, please provide using --uri, the config or environment variable PYICEBERG_CATALOG__DEFAULT__URI

Is very misleading..

It comes on many ways such as pyiceberg list --help, and usually --help is the flag that you use to look help for a command, like examples of the usage of the command.

The need of having the --catalog hive there, even if you would check the help, is counter intuitive. Example pyiceberg --catalog hive list --help works, but pyiceberg list --help does not.

[kevinjqliu]

Thanks @lk-1984 for the detailed debugging session :)

I agree that the CLI should provide more user friendly warnings regarding catalog selection and override. I've opened #1784 to track this issue