Merge pull request OSeMOSYS#190 from trevorb1/cplex

Read in CPLEX Results

docs/examples.rst

@@ -6,7 +6,7 @@ Examples
 
 This page will present examples to show the full functionality of ``otoole``. It will
 walk through the ``convert``, ``results``, ``setup``, ``viz`` and ``validate``
-functionality in seperate simple use cases.
+functionality in separate simple use cases.
 
 .. NOTE::
     To follow these examples, clone the Simplicity_ repository and run all commands
@@ -34,12 +34,12 @@ abbreviated instructions are shown below
 
 To install GLPK on **Linux**, run the command::
 
-    sudo apt-get update
-    sudo apt-get install glpk glpk-utils
+    $ sudo apt-get update
+    $ sudo apt-get install glpk glpk-utils
 
 To install GLPK on **Mac**, run the command::
 
-    brew install glpk
+    $ brew install glpk
 
 To install GLPK on **Windows**, follow the instructions on the
 `GLPK Website`_. Be sure to add GLPK to
@@ -48,7 +48,7 @@ your environment variables after installation
 Alternatively, if you use Anaconda_ to manage
 your Python packages, you can install GLPK via the command::
 
-    conda install -c conda-forge glpk
+    $ conda install -c conda-forge glpk
 
 2. Test the GLPK install
 ~~~~~~~~~~~~~~~~~~~~~~~~
@@ -58,6 +58,9 @@ Once installed, you should be able to call the ``glpsol`` command::
     GLPSOL: GLPK LP/MIP Solver, v4.65
     No input problem file specified; try glpsol --help
 
+.. TIP::
+    See the `GLPK Wiki`_ for more information on the ``glpsol`` command
+
 3. Install CBC
 ~~~~~~~~~~~~~~
 
@@ -67,19 +70,19 @@ instructions are shown below
 
 To install CBC on **Linux**, run the command::
 
-    sudo apt-get install coinor-cbc coinor-libcbc-dev
+    $ sudo apt-get install coinor-cbc coinor-libcbc-dev
 
 To install CBC on **Mac**, run the command::
 
-    brew install coin-or-tools/coinor/cbc
+    $ brew install coin-or-tools/coinor/cbc
 
 To install CBC on **Windows**, follow the install instruction on the CBC_
 website.
 
 Alternatively, if you use Anaconda_ to manage
 your Python packages, you can install CBC via the command::
 
-    conda install -c conda-forge coincbc
+    $ conda install -c conda-forge coincbc
 
 4. Test the CBC install
 ~~~~~~~~~~~~~~~~~~~~~~~
@@ -96,122 +99,103 @@ Once installed, you should be able to directly call CBC::
 
 You can exit the solver by typing ``quit``
 
-Data Conversion with CSVs
--------------------------
+Input Data Conversion
+---------------------
 
 Objective
 ~~~~~~~~~
 
-Use a folder of CSV data to build and solve an OSeMOSYS model with CBC_. Generate
-the full suite of OSeMOSYS results.
-
-1. ``otoole`` Convert
-~~~~~~~~~~~~~~~~~~~~~
-We first want to convert the folder of Simplicity_ CSVs into
-an OSeMOSYS datafile called ``simplicity.txt``::
-
-    $ otoole convert csv datafile data simplicity.txt config.yaml
-
-2. Build the Model
-~~~~~~~~~~~~~~~~~~~
-Use GLPK_ to build the model and save it as ``simplicity.lp``::
+Convert input data between CSV, Excel, and GNU MathProg data formats.
 
-    $ glpsol -m OSeMOSYS.txt -d simplicity.txt --wlp simplicity.lp --check
+1. Clone ``Simplicity``
+~~~~~~~~~~~~~~~~~~~~~~~
+If not already done so, clone the Simplicity_ repository::
 
-.. TIP::
-    See the `GLPK Wiki`_ for more information on the ``glpsol`` command
+    $ git clone https://github.com/OSeMOSYS/simplicity.git
+    $ cd simplicity
 
-3. Solve the Model
-~~~~~~~~~~~~~~~~~~
-Use CBC_ to solve the model and save the solution file as ``simplicity.sol``::
+.. NOTE::
+    Further information on the ``config.yaml`` file is in the :ref:`template-setup` section
 
-    $ cbc simplicity.lp solve -solu simplicity.sol
+2. Convert CSV data into MathProg data
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Convert the folder of Simplicity_ CSVs (``data/``) into an OSeMOSYS datafile called ``simplicity.txt``::
 
-4. Generate the full set of results
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Use ``otoole``'s ``result`` package to generate the results file::
+    $ otoole convert csv datafile data simplicity.txt config.yaml
 
-    $ otoole results cbc csv simplicity.sol results datafile simplicity.txt config.yaml
+3. Convert MathProg data into Excel Data
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Convert the new Simplicity_ datafile (``simplicity.txt``) into Excel data called ``simplicity.xlsx``::
 
-5. View Results
-~~~~~~~~~~~~~~~
-Results are now viewable in the files ``results/*.csv``
+    $ otoole convert datafile excel simplicity.txt simplicity.xlsx config.yaml
 
 .. TIP::
-    Before moving onto the next section, remove all the generated files::
+    Excel workbooks are an easy way for humans to interface with OSeMOSYS data!
+
+4. Convert Excel Data into CSV data
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Convert the new Simplicity_ excel data (``simplicity.xlsx``) into a folder of CSV data
+called ``simplicity/``. Note that this data will be the exact same as the original CSV data folder (``data/``)::
 
-        $ rm simplicity.lp simplicity.sol simplicity.txt results/*
+    $ otoole convert excel csv simplicity.xlsx simplicity config.yaml
 
-Data Conversion with Excel
---------------------------
+Process Solutions from Different Solvers
+----------------------------------------
 
 Objective
 ~~~~~~~~~
 
-Use an excel worksheet to build and solve an OSeMOSYS model with CBC.
+Process solutions from GLPK_, CBC_, Gurobi_, and CPLEX_. This example assumes
+you have an existing GNU MathProg datafile called ``simplicity.txt`` (from the
+previous example).
 
-1. Create the Excel Workbook
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Use the example CSV data to create an Excel Workbook using ``otoole convert``::
-
-    $ otoole convert csv excel data simplicity.xlsx config.yaml
+1. Process a solution from GLPK
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Use GLPK_ to build the model, save the problem as ``simplicity.glp``, solve the model, and
+save the solution as ``simplicity.sol``. Use otoole to create a folder of CSV results called ``results-glpk/``.
+When processing solutions from GLPK, the model file (``*.glp``) must also be passed::
 
-Excel workbooks are an easy way for humans to interface with OSeMOSYS data!
+    $ glpsol -m OSeMOSYS.txt -d simplicity.txt --wglp simplicity.glp --write simplicity.sol
 
-2. Create the MathProg datafile
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Next, we want to convert the excel workbook (``simplicity.xlsx``) into
-an OSeMOSYS datafile (``simplicity.txt``)::
+    $ otoole results glpk csv simplicity.sol results-glpk datafile simplicity.txt config.yaml --glpk_model simplicity.glp
 
-    $ otoole convert excel datafile simplicity.xlsx simplicity.txt config.yaml
+.. NOTE::
+   By default, MathProg OSeMOSYS models will write out folder of CSV results to a ``results/``
+   directory if solving via GLPK. However, using ``otoole`` allows the user to programmatically access results
+   and control read/write locations
 
-3. Build the Model
-~~~~~~~~~~~~~~~~~~
-Use GLPK_ to build the model and save it as ``simplicity.lp``::
+2. Process a solution from CBC
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Use GLPK_ to build the model and save the problem as ``simplicity.lp``. Use CBC_ to solve the model and
+save the solution as ``simplicity.sol``. Use otoole to create a folder of CSV results called ``results/`` from the solution file::
 
     $ glpsol -m OSeMOSYS.txt -d simplicity.txt --wlp simplicity.lp --check
 
-4. Solve the Model
-~~~~~~~~~~~~~~~~~~
-Use CBC_ to solve the model and save the solution file as ``simplicity.sol``::
-
     $ cbc simplicity.lp solve -solu simplicity.sol
 
-5. Generate the selected results
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Use ``otoole``'s ``result`` package to generate the result CSVs::
-
-    $ otoole results cbc csv simplicity.sol results datafile simplicity.txt config.yaml
-
-Data Processing with GLPK
--------------------------
+    $ otoole results cbc csv simplicity.sol results csv data config.yaml
 
-Objective
-~~~~~~~~~
+3. Process a solution from Gurobi
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Use GLPK_ to build the model and save the problem as ``simplicity.lp``. Use Gurobi_ to solve the model and
+save the solution as ``simplicity.sol``. Use otoole to create a folder of CSV results called ``results/`` from the solution file::
 
-Build and solve a model using GLPK and otoole
+    $ glpsol -m OSeMOSYS.txt -d simplicity.txt --wlp simplicity.lp --check
 
-1. Build the solve the model using GLPK
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Use GLPK_ to build the model, save the problem as ``simplicity.glp``, solve the model,
-and save the solution as ``simplicity.sol```::
+    $ gurobi_cl ResultFile=simplicity.sol simplicity.lp
 
-    $ glpsol -m OSeMOSYS.txt -d simplicity.txt --wglp simplicity.glp --write simplicity.sol
+    $ otoole results gurobi csv simplicity.sol results csv data config.yaml
 
-2. Use otoole to process the solution in CSVs
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Use ``otoole``'s ``results`` command to transform the soltuion file into a folder of CSVs
-under the directory ``results-glpk``. When processing solutions from GLPK, the model file (``*.glp``)
-must also be passed::
+4. Process a solution from CPLEX
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Use GLPK_ to build the model and save the problem as ``simplicity.lp``. Use CPLEX_ to solve the model and
+save the solution as ``simplicity.sol``. Use otoole to create a folder of CSV results called ``results/`` from the solution file::
 
-    $ otoole results glpk csv simplicity.sol results-glpk datafile simplicity.txt config.yaml --glpk_model simplicity.glp
+    $ glpsol -m OSeMOSYS.txt -d simplicity.txt --wlp simplicity.lp --check
 
-.. NOTE::
-   By default, MathProg OSeMOSYS models will write out folder of CSV results to a ``results/``
-   directory if solving via GLPK. However, for programatically accessing results, using ``otoole``
-   to control the read/write location, and for supporting future implementations of OSeMOSYS,
-   using ``otoole`` can be benifical.
+    $ cplex -c "read simplicity.lp" "optimize" "write simplicity.sol"
 
+    $ otoole results cplex csv simplicity.sol results csv data config.yaml
 
 Model Visualization
 -------------------
@@ -238,6 +222,8 @@ displayed
 
 .. image:: _static/simplicity_res.png
 
+.. _template-setup:
+
 Template Setup
 --------------
 
@@ -284,13 +270,12 @@ horizon. For example, if the model horizon is from 2020 to 2050, the
 
 .. NOTE::
    While this step in not technically required, by filling out the years in
-   CSV format, ``otoole`` will pivot all the Excel sheets on the years
-   during the conversion process. This will save significant formatting time!
+   CSV format ``otoole`` will pivot all the Excel sheets on these years.
+   This will save significant formatting time!
 
 4. Convert the CSV Template Data
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-To convert the template CSV data into Excel formatted data, run the following
-``convert`` command::
+Convert the template CSV data into Excel formatted data::
 
     $ otoole convert csv excel template_data template.xlsx template_config.yaml
 
@@ -500,3 +485,4 @@ will also flag it as an isolated fuel. This means the fuel is unconnected from t
 .. _CBC: https://github.com/coin-or/Cbc
 .. _CPLEX: https://www.ibm.com/products/ilog-cplex-optimization-studio/cplex-optimizer
 .. _Anaconda: https://www.anaconda.com/
+.. _Gurobi: https://www.gurobi.com/

docs/functionality.rst

@@ -89,25 +89,26 @@ so as to speed up the model matrix generation and solution times.
 ``otoole results``
 ~~~~~~~~~~~~~~~~~~
 
-The ``results`` command creates a folder of CSV result files from a CBC_, CLP_,
+The ``results`` command creates a folder of CSV result files from a GLPK_, CBC_, CLP_,
 Gurobi_ or CPLEX_ solution file together with the input data::
 
     $ otoole results --help
-    usage: otoole results [-h] [--write_defaults]
+    usage: otoole results [-h] [--glpk_model GLPK_MODEL] [--write_defaults]
                         {cbc,cplex,gurobi} {csv} from_path to_path {csv,datafile,excel} input_path config
 
     positional arguments:
-    {cbc,cplex,glpk,gurobi}    Result data format to convert from
-    {csv}                 Result data format to convert to
-    from_path             Path to file or folder to convert from
-    to_path               Path to file or folder to convert to
-    {csv,datafile,excel}  Input data format
-    input_path            Path to input_data
-    config                Path to config YAML file
+    {cbc,cplex,glpk,gurobi}  Result data format to convert from
+    {csv}                    Result data format to convert to
+    from_path                Path to file or folder to convert from
+    to_path                  Path to file or folder to convert to
+    {csv,datafile,excel}     Input data format
+    input_path               Path to input_data
+    config                   Path to config YAML file
 
     optional arguments:
-    -h, --help            show this help message and exit
-    --write_defaults      Writes default values
+    -h, --help                show this help message and exit
+    --glpk_model GLPK_MODEL   GLPK model file required for processing GLPK results
+    --write_defaults          Writes default values
 
 .. versionadded:: v1.0.0
     The ``config`` positional argument is now required

src/otoole/convert.py

@@ -31,7 +31,7 @@ def read_results(
     input_path: str,
     glpk_model: str = None,
 ) -> Tuple[Dict[str, pd.DataFrame], Dict[str, float]]:
-    """Read OSeMOSYS results from CBC, GLPK or Gurobi results files
+    """Read OSeMOSYS results from CBC, GLPK, Gurobi, or CPLEX results files
 
     Arguments
     ---------