serialization of columns added into the definition of the table

[matteocacciola]

• Closes Columns not added during the serialization of the DataFrame #1714.
• Tests added and passed if fixing a bug or adding a new feature.
• All code checks passed.

---

Important

Adds column serialization to DataframeSerializer.serialize() and updates tests and documentation accordingly.

• Behavior:
  • DataframeSerializer.serialize() in dataframe_serializer.py now includes column serialization in the table definition.
  • is_expression_valid() in semantic_layer_schema.py now returns Optional[str] to handle None values.
• Testing:
  • Updates test_dataframe_serializer.py to test new column serialization format.
  • Adds new test commands make test_core, make test_extensions, and make test-coverage in CONTRIBUTING.md.

^{This description was created by for 2d97886. It will automatically update as commits are pushed.}

[ellipsis-dev]

👍 Looks good to me! Reviewed everything up to 2d97886 in 43 seconds

More details

• Looked at 107 lines of code in 4 files
• Skipped 1 files when reviewing.
• Skipped posting 10 drafted comments based on config settings.

1. CONTRIBUTING.md:65

• Draft comment:
  Updated test commands (make test_all, test_core, test_extensions, test-coverage) are clearly documented. Ensure future updates maintain consistent naming conventions throughout.
• Reason this comment was not posted:
  Confidence changes required: 0% <= threshold 50%
  None

2. pandasai/data_loader/semantic_layer_schema.py:64

• Draft comment:
  In the is_expression_valid validator, adding an explicit check for None is good. Consider if empty strings should be allowed or handled similarly.
• Reason this comment was not posted:
  Confidence changes required: 0% <= threshold 50%
  None

3. pandasai/helpers/dataframe_serializer.py:31

• Draft comment:
  Serialization now includes a 'columns' attribute with JSON output. This implementation is clear; ensure that any future changes to Column model are reflected in the serializer tests.
• Reason this comment was not posted:
  Confidence changes required: 0% <= threshold 50%
  None

4. tests/unit_tests/helpers/test_dataframe_serializer.py:9

• Draft comment:
  Test expectations updated with the new columns JSON attribute. Confirm the nested double quotes are rendered as expected in different environments.
• Reason this comment was not posted:
  Confidence changes required: 0% <= threshold 50%
  None

5. CONTRIBUTING.md:65

• Draft comment:
  Good update of test commands. Ensure that any command-line updates in documentation are synced with the makefile targets.
• Reason this comment was not posted:
  Confidence changes required: 0% <= threshold 50%
  None

6. pandasai/data_loader/semantic_layer_schema.py:64

• Draft comment:
  The validator now returns None if expression is None. Consider updating the docstring (or adding a comment) to clarify that a missing expression bypasses SQL parsing.
• Reason this comment was not posted:
  Confidence changes required: 0% <= threshold 50%
  None

7. pandasai/helpers/dataframe_serializer.py:31

• Draft comment:
  New logic adds a 'columns' attribute with JSON-dumped column definitions. Verify that HTML attribute escaping is robust if column descriptions may include quotes or special characters.
• Reason this comment was not posted:
  Confidence changes required: 0% <= threshold 50%
  None

8. tests/unit_tests/helpers/test_dataframe_serializer.py:8

• Draft comment:
  The expected output embeds JSON inside an HTML attribute. This literal comparison may be fragile if key ordering changes. Consider parsing the JSON portion and comparing as objects to make tests more robust.
• Reason this comment was not posted:
  Confidence changes required: 50% <= threshold 50%
  None

9. CONTRIBUTING.md:56

• Draft comment:
  Typo found: On line 56, "We usee codespell..." should be corrected to "We use codespell...".
• Reason this comment was not posted:
  Comment was not on a location in the diff, so it can't be submitted as a review comment.

10. pandasai/helpers/dataframe_serializer.py:8

• Draft comment:
  Minor naming inconsistency: The class is named 'DataframeSerializer' (with a lowercase 'f'), whereas the type hints and commonly used naming conventions (e.g., DataFrame) suggest it might be more consistent to name it 'DataFrameSerializer'. This is a trivial cosmetic issue.
• Reason this comment was not posted:
  Comment was not on a location in the diff, so it can't be submitted as a review comment.

Workflow ID: wflow_9zWS9G4iItB46wo8

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

[matteocacciola]

Hello @ArslanSaleem @gdcsinaptik any chance to consider this PR?