Add parameter correlations table/plot (#146)

* Reorganize tmp/ directory

* Implement parameter correlations table/plot

* Enhance correlation heatmap with threshold and precision parameters for labels

* Refine documentation and examples

* Refactor example notebooks to improve clarity

* Temporarily lower minimum coverage percentage requirement from 75 to 70

* Remove show_diagonal and triangle from plot_param_correlations

* Fix PlotterBase.plot_correlation_heatmap lint errors

* Fix future annotations, dead code, and **kwargs in display

* Clean up

Author: AndrewSazonov

docs/architecture/sequential_fitting_design.md

@@ -1120,7 +1120,7 @@ propagation, diffrn callback, precondition validation.
 **Implemented:** `Plotter.plot_param_series()` resolves CSV vs snapshots
 automatically via the project reference.
 `Plotter._plot_param_series_from_csv()` reads CSV via pandas.
-`Plotter._plot_param_series_from_snapshots()` preserves backward
+`Plotter.plot_param_series_from_snapshots()` preserves backward
 compatibility for `fit()` single-mode (no CSV yet). Axis labels derived
 from live descriptor objects.
 

docs/docs/tutorials/ed-1.ipynb

@@ -3,7 +3,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "b1e6b328",
+   "id": "ab0fa7f5",
    "metadata": {
     "tags": [
      "hide-in-docs"
@@ -26,23 +26,20 @@
    "source": [
     "# Structure Refinement: LBCO, HRPT\n",
     "\n",
-    "This minimalistic example is designed to show how Rietveld refinement\n",
-    "can be performed when both the crystal structure and experiment\n",
-    "parameters are defined using CIF files.\n",
+    "This basic example is designed to show how Rietveld refinement can be\n",
+    "performed when both the crystal structure and experiment parameters\n",
+    "are defined using CIF files.\n",
     "\n",
     "For this example, constant-wavelength neutron powder diffraction data\n",
     "for La0.5Ba0.5CoO3 from HRPT at PSI is used.\n",
     "\n",
-    "It does not contain any advanced features or options, and includes no\n",
-    "comments or explanations—these can be found in the other tutorials.\n",
-    "Default values are used for all parameters if not specified. Only\n",
-    "essential and self-explanatory code is provided.\n",
-    "\n",
     "The example is intended for users who are already familiar with the\n",
-    "EasyDiffraction library and want to quickly get started with a simple\n",
-    "refinement. It is also useful for those who want to see what a\n",
-    "refinement might look like in code. For a more detailed explanation of\n",
-    "the code, please refer to the other tutorials."
+    "EasyDiffraction library and want to quickly get started with a basic\n",
+    "refinement.\n",
+    "\n",
+    "It is also useful for those who want to see how constraints can be\n",
+    "applied to highly correlated parameters. For a more detailed\n",
+    "explanation of the code, please refer to the other tutorials."
    ]
   },
   {
@@ -147,7 +144,7 @@
    "id": "11",
    "metadata": {},
    "source": [
-    "## Step 4: Perform Analysis (cryspy)"
+    "## Step 4: Perform Analysis (no constraints)"
    ]
   },
   {
@@ -157,18 +154,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Define aliases and constraints for refinement. This is necessary to\n",
-    "# properly refine the isotropic displacement parameters of La and Ba,\n",
-    "# which are correlated due to their shared Wyckoff position.\n",
-    "project.analysis.aliases.create(\n",
-    "    label='biso_La',\n",
-    "    param=project.structures['lbco'].atom_sites['La'].b_iso,\n",
-    ")\n",
-    "project.analysis.aliases.create(\n",
-    "    label='biso_Ba',\n",
-    "    param=project.structures['lbco'].atom_sites['Ba'].b_iso,\n",
-    ")\n",
-    "project.analysis.constraints.create(expression='biso_Ba = biso_La')"
+    "# Start refinement. All parameters, which have standard uncertainties\n",
+    "# in the input CIF files, are refined by default.\n",
+    "project.analysis.fit()"
    ]
   },
   {
@@ -178,9 +166,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Start refinement. All parameters, which have standard uncertainties\n",
-    "# in the input CIF files, are refined by default.\n",
-    "project.analysis.fit()"
+    "# Show fit results summary\n",
+    "project.analysis.display.fit_results()"
    ]
   },
   {
@@ -190,19 +177,16 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Show fit results summary\n",
-    "project.analysis.display.fit_results()"
+    "# Show parameter correlations\n",
+    "project.plotter.plot_param_correlations()"
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
+   "cell_type": "markdown",
    "id": "15",
    "metadata": {},
-   "outputs": [],
    "source": [
-    "# Show defined experiment names\n",
-    "project.experiments.show_names()"
+    "## Step 5: Perform Analysis (with constraints)"
    ]
   },
   {
@@ -212,16 +196,34 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Plot measured vs. calculated diffraction patterns\n",
-    "project.plotter.plot_meas_vs_calc(expt_name='hrpt', show_residual=True)"
+    "# As can be seen from the parameter-correlation plot, the isotropic\n",
+    "# displacement parameters of La and Ba are highly correlated. Because\n",
+    "# La and Ba share the same mixed-occupancy site, their contributions to\n",
+    "# the neutron diffraction pattern are difficult to separate, especially\n",
+    "# since their coherent scattering lengths are not very different.\n",
+    "# Therefore, it is necessary to constrain them to be equal. First we\n",
+    "# define aliases and then use them to create a constraint.\n",
+    "project.analysis.aliases.create(\n",
+    "    label='biso_La',\n",
+    "    param=project.structures['lbco'].atom_sites['La'].b_iso,\n",
+    ")\n",
+    "project.analysis.aliases.create(\n",
+    "    label='biso_Ba',\n",
+    "    param=project.structures['lbco'].atom_sites['Ba'].b_iso,\n",
+    ")\n",
+    "project.analysis.constraints.create(expression='biso_Ba = biso_La')"
    ]
   },
   {
-   "cell_type": "markdown",
+   "cell_type": "code",
+   "execution_count": null,
    "id": "17",
    "metadata": {},
+   "outputs": [],
    "source": [
-    "## Step 5: Perform Analysis (crysfml)"
+    "# Start refinement. All parameters, which have standard uncertainties\n",
+    "# in the input CIF files, are refined by default.\n",
+    "project.analysis.fit()"
    ]
   },
   {
@@ -231,9 +233,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Change calculation engine from 'cryspy' to 'crysfml'\n",
-    "project.experiments['hrpt'].show_supported_calculator_types()\n",
-    "project.experiments['hrpt'].calculator_type = 'crysfml'"
+    "# Show fit results summary\n",
+    "project.analysis.display.fit_results()"
    ]
   },
   {
@@ -243,8 +244,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Start refinement\n",
-    "project.analysis.fit()"
+    "# Show parameter correlations\n",
+    "project.plotter.plot_param_correlations()"
    ]
   },
   {
@@ -254,8 +255,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Show fit results summary\n",
-    "project.analysis.display.fit_results()"
+    "# Show defined experiment names\n",
+    "project.experiments.show_names()"
    ]
   },
   {

docs/docs/tutorials/ed-1.py

@@ -1,23 +1,20 @@
 # %% [markdown]
 # # Structure Refinement: LBCO, HRPT
 #
-# This minimalistic example is designed to show how Rietveld refinement
-# can be performed when both the crystal structure and experiment
-# parameters are defined using CIF files.
+# This basic example is designed to show how Rietveld refinement can be
+# performed when both the crystal structure and experiment parameters
+# are defined using CIF files.
 #
 # For this example, constant-wavelength neutron powder diffraction data
 # for La0.5Ba0.5CoO3 from HRPT at PSI is used.
 #
-# It does not contain any advanced features or options, and includes no
-# comments or explanations—these can be found in the other tutorials.
-# Default values are used for all parameters if not specified. Only
-# essential and self-explanatory code is provided.
-#
 # The example is intended for users who are already familiar with the
-# EasyDiffraction library and want to quickly get started with a simple
-# refinement. It is also useful for those who want to see what a
-# refinement might look like in code. For a more detailed explanation of
-# the code, please refer to the other tutorials.
+# EasyDiffraction library and want to quickly get started with a basic
+# refinement.
+#
+# It is also useful for those who want to see how constraints can be
+# applied to highly correlated parameters. For a more detailed
+# explanation of the code, please refer to the other tutorials.
 
 # %% [markdown]
 # ## Import Library
@@ -55,12 +52,32 @@
 project.experiments.add_from_cif_path(expt_path)
 
 # %% [markdown]
-# ## Step 4: Perform Analysis (cryspy)
+# ## Step 4: Perform Analysis (no constraints)
+
+# %%
+# Start refinement. All parameters, which have standard uncertainties
+# in the input CIF files, are refined by default.
+project.analysis.fit()
+
+# %%
+# Show fit results summary
+project.analysis.display.fit_results()
+
+# %%
+# Show parameter correlations
+project.plotter.plot_param_correlations()
+
+# %% [markdown]
+# ## Step 5: Perform Analysis (with constraints)
 
 # %%
-# Define aliases and constraints for refinement. This is necessary to
-# properly refine the isotropic displacement parameters of La and Ba,
-# which are correlated due to their shared Wyckoff position.
+# As can be seen from the parameter-correlation plot, the isotropic
+# displacement parameters of La and Ba are highly correlated. Because
+# La and Ba share the same mixed-occupancy site, their contributions to
+# the neutron diffraction pattern are difficult to separate, especially
+# since their coherent scattering lengths are not very different.
+# Therefore, it is necessary to constrain them to be equal. First we
+# define aliases and then use them to create a constraint.
 project.analysis.aliases.create(
     label='biso_La',
     param=project.structures['lbco'].atom_sites['La'].b_iso,
@@ -81,28 +98,12 @@
 project.analysis.display.fit_results()
 
 # %%
-# Show defined experiment names
-project.experiments.show_names()
-
-# %%
-# Plot measured vs. calculated diffraction patterns
-project.plotter.plot_meas_vs_calc(expt_name='hrpt', show_residual=True)
-
-# %% [markdown]
-# ## Step 5: Perform Analysis (crysfml)
-
-# %%
-# Change calculation engine from 'cryspy' to 'crysfml'
-project.experiments['hrpt'].show_supported_calculator_types()
-project.experiments['hrpt'].calculator_type = 'crysfml'
-
-# %%
-# Start refinement
-project.analysis.fit()
+# Show parameter correlations
+project.plotter.plot_param_correlations()
 
 # %%
-# Show fit results summary
-project.analysis.display.fit_results()
+# Show defined experiment names
+project.experiments.show_names()
 
 # %%
 # Plot measured vs. calculated diffraction patterns

docs/docs/tutorials/ed-10.ipynb

@@ -3,7 +3,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "307e00fa",
+   "id": "dc88a9ac",
    "metadata": {
     "tags": [
      "hide-in-docs"

docs/docs/tutorials/ed-11.ipynb

@@ -3,7 +3,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "2ab104e6",
+   "id": "9736040b",
    "metadata": {
     "tags": [
      "hide-in-docs"

docs/docs/tutorials/ed-12.ipynb

@@ -3,7 +3,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "f188a46a",
+   "id": "44960a4c",
    "metadata": {
     "tags": [
      "hide-in-docs"

docs/docs/tutorials/ed-13.ipynb

@@ -3,7 +3,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "be0cfa00",
+   "id": "1ac207fe",
    "metadata": {
     "tags": [
      "hide-in-docs"
@@ -2647,7 +2647,7 @@
  ],
  "metadata": {
   "jupytext": {
-   "cell_metadata_filter": "title,tags,-all",
+   "cell_metadata_filter": "tags,title,-all",
    "main_language": "python",
    "notebook_metadata_filter": "-all"
   }

docs/docs/tutorials/ed-14.ipynb

@@ -3,7 +3,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "f6f4b6a6",
+   "id": "86c9f966",
    "metadata": {
     "tags": [
      "hide-in-docs"

docs/docs/tutorials/ed-15.ipynb

@@ -3,7 +3,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "ceaaff89",
+   "id": "c631fd19",
    "metadata": {
     "tags": [
      "hide-in-docs"

docs/docs/tutorials/ed-16.ipynb

@@ -3,7 +3,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "c342992c",
+   "id": "d0fcc613",
    "metadata": {
     "tags": [
      "hide-in-docs"