Add profiling & benchmarks for Indexset and Parameter -- clean
#150
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Initial Release
* Update installation in README * Update installation in DEVELOPING * Add dynamic versioning * Add publish workflow * Update poetry publish version * Add info about releasing and version number
* Migrate candidate 0.2.0 version * delete version test
* Try enabling postgres * Explicitly install libpq-dev * Remove libpq-dev since it's already installed * Switch psycopg2 to psycopg2-binary * Revert poetry.lock * Try installing python-dev * Fix apt-get install command * Add alternative publish workflow * Add dynamic versioning * Set up alternative publishing workflow for repositories * Fix faulty publish workflow * Enable using trusted publisher * Make alternative publish workflow proper * Add missing permission setting
* Merge current state of ixmp4-dev into ixmp4 * Files have been selected based on a diff of the two repos * Where changes were detected, ixmp4-dev version was used * Remove falsely added dynamic-versioning dependencies
* Add isort as part of dev dependencies * Apply isort to all files * Update anyio and pandas dependencies * Specify import location of Backend to resolve circular import
* Update GHA to run on python 3.10 and 3.11 * Introduce GHA checks for black and isort on PRs * Update badges to show python 3.11 support * Run GHA only on necessary python versions * Combine code style checks in lint.yaml * Update README.md (#15) * Combine updated python badges * Add build in dev dependency group --------- Co-authored-by: Daniel Huppmann <[email protected]>
Co-authored-by: Philip Hackstock <[email protected]>
This reverts commit 809be62.
* regress tests * fix wrong types for *like lookups supplied
* adjust dependency psycopg2->psycopg[binary] * require postgresql+psycopg:// in dsn * replace old dsn prefix to keep .toml backwards compatibility * update docs
* Adapt to pydantic v2.4, fastapi 0.103 * Let pydantic return a SchemaError instead of InconsistentIamcType * Use pandera.DataFrameModel instead of pandera.SchemaModel * Clean up changes in API layer * Make creation information optional in iamc/variable * Make id and name required for unit * Revert bulk_* changes to use json_encoder of api.DataFrame again * NOTE: this option will be deprecated in the future, * pydantic serialization decorators should be used instead * Clean up changes in data/db to enable tests again * Revert use of Annotated in iamc/datapoint/filter * Make sqla_model a proper ClassVar (rather than private attr) * Set field.alias only when it does not equal field.name * Update version of ruff * Set field alias in Field directly * Use httpx-preferred content to send bytes/raw content/json st * Use alias for model_ fields that conflict with pydantic protected namespace * Use variable rather than unmapped class reference to resolve SAwarning * Delete unused import * Bump mypy and poetry versions * Use pre-commit over lint-GHA * Runs mypy, black, ruff * NOTE: needs to be run in the repo s.t. poetry detects the correct venv * Remove separate lint.yaml action * Bump sphinx version * Update docstrings according to new sphinx version * Delete duplicate docs files * TEMPORARY Use own fork for sphinxcontrib-openapi workaround * Bump dask,sqlalchemy,pre-commit versions * restore filter framework api * fix union types with filter subojects * fix code style * fix formatting error * Remove outdated comments * Update various things: * Bump fastapi,black,ruff,mypy versions * Update pre-commit-config accordingly * Update doc/source/openapi-v1.json * Include pre-commit, ruff, etc in DEVELOPING.md --------- Co-authored-by: Max Wolschlager <[email protected]>
* Introduce optimization.Variable * Make constrained_to_indexset optional for scalar Variables * Fix and test variable list and tabulate for specific runs * Fix name of Variable table * Introduce variable.remove_data() * Remove superfluous session.add() for variable * Use own errors for Variable
* Introduce optimization.Equation * Fix and test equation list and tabulate for specific runs * Introduce equation.remove_data() * Remove superfluous session.add() for equation * Use own errors for Equation
* try to use NullPool to fix idle connection issue * omit session.rollback * add engine.dispose again * Bump versions of pre-commit tools * Delete old ixmp version of transport tutorial --------- Co-authored-by: Fridolin Glatter <[email protected]>
* Enable Codecov uploading and reporting on CI * Remove Codecov terminal output
* implement SqlaEventHandler.pause ctx manager * pause event handlers during re-setting of default version * adjust tests
* Run whole test suite also on pushes to main * Remove outdated build-docs workflow
* Add new DB data types and imports * Refactor indexset.add_data() * Introduce new IndexSetData table to normalize data * Rename .elements to .add for consistency * Adapt tests to refactoring * Cast types using numpy for efficiency * Reduce number of lines needing tests * Make data loading during tabulate() optional * Use bulk insert for IndexSetData * Use a normal property for `.data` * Remove not-required parameter * Add all missing and new db migrations
* Add support for Python 3.13 * Adapt badge on readme page * Adapt test for 3.12 and 3.13 * Bump various dependencies to support 3.13 in lock file
* Complete type hints * Bump mypy to latest release version * Enable mypy for docs and tests * Note reasons for type: ignore for future reference * DRY enumerate kwargs * DRY more type hints using TypeAlias * Exclude if TYPE_CHECKING from coverage * Add mypy badge to README * Make scalar.unit non-optional * Bump pandas version in poetry.lock
* Add abstraction layer for core optimization reusability * Add annotations
Base automatically changed from
fix/avoid-superfluous-db-calls-from-core
to
main
January 16, 2025 14:07
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #150 +/- ##
=======================================
- Coverage 86.9% 86.9% -0.1%
=======================================
Files 230 230
Lines 8156 8144 -12
=======================================
- Hits 7095 7082 -13
- Misses 1061 1062 +1
|
This was referenced Jan 16, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In #148, we're adding a very large data file:
big/parameterdata.csvhas almost 100MB. In order to avoid bloating the git history of this repo, this PR was created loosely following the procedure outlined here: I created a clean clone of ixmp4 and rewrote its history to remove the very large file from it entirely. Then I merged the resulting branch back into a branch freshly started off ofmain, and I'm using this to open this PR.The preview shows 116 commits, but it's actually just one, merging the clean branch to the new one. The files changed seems reasonable, though, so I figure we should be fine by squashing these apparent commits when merging.