final updates for PCSE 5.4

- Update to NASAPowerDP to properly take missing values into account
- Update of documentation to take Py3 compatibility into account

Author: ajwdewit

README.rst

@@ -23,9 +23,9 @@ in pure python code which makes it more flexible, easier to modify and extensibl
 allowing easy interfacing with databases, graphical user interfaces, visualization
 tools and numerical/statistical packages. PCSE has several interesting features:
 
-* Implementation in pure python and the core system has a small number of dependencies outside the python standard
-  library. Most of these can be automatically installed from the Python Package Index (PyPI) (`SQLAlchemy`, `PyYAML`,
-  `tabulate`, `xlwt`, `xlrd`) although some additional modules rely on `NumPy`.
+* Implementation in pure python with dependencies only on popular packages available from
+  the Python Package Index (PyPI) (`SQLAlchemy`, `PyYAML`, `pandas`, `xlwt`, `xlrd`,
+  `requests` and `numpy`)
 
 * Modular design allowing you to add or change components relatively quickly with
   a simple but powerful approach to communicate variables between modules.

doc/conf.py

@@ -103,7 +103,8 @@
 
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
-html_theme = 'sphinxdoc'
+#html_theme = 'sphinxdoc'
+html_theme = "sphinx_rtd_theme"
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the

doc/downloads/py2_pcse.yml

@@ -1,30 +1,19 @@
 name: py2_pcse
 dependencies:
-- alabaster
-- babel
-- colorama
-- docutils
-- imagesize
-- jinja2
-- markupsafe
-- mkl
+- sqlalchemy
+- pyyaml
 - numpy
-- pip
-- pygments
+- pandas
 - python=2.7
 - pytz
 - pyyaml
 - setuptools
-- six
-- snowballstemmer
 - sphinx
 - sphinx_rtd_theme
-- sqlalchemy
-- vs2008_runtime
-- wheel
 - xlrd
 - xlwt
+- requests
+- ipython
+- jupyter
 - pip:
-  - sphinx-rtd-theme==0.1.9
-  - tabulate==0.7.5
-
+  - traitlets-pcse

doc/downloads/py2_pcse_full.yml

@@ -1,9 +1,10 @@
 name: py3_pcse
+dependencies:
 - sqlalchemy
 - pyyaml
 - numpy
 - pandas
-- python=3
+- python=3.6
 - pytz
 - pyyaml
 - setuptools
@@ -12,6 +13,8 @@ name: py3_pcse
 - xlrd
 - xlwt
 - requests
+- ipython
+- jupyter
 - pip:
   - traitlets-pcse
 

doc/downloads/py3_pcse.yml

@@ -5,9 +5,8 @@ Installing PCSE
 Requirements and dependencies
 =============================
 
-PCSE is being developed on Ubuntu Linux 14.04 and Windows 7 using Python 2.7.11 and is known to work with
-the 3.x series (using the 2to3 tool). As Python is a platform independent language, PCSE
-works equally well on Linux, Windows or Mac OSX.
+PCSE is being developed on Ubuntu Linux 14.04 and Windows 7 using Python 2.7.14 and python 3.6.5 .
+As Python is a platform independent language, PCSE works equally well on Linux, Windows or Mac OSX.
 Before installing PCSE, Python itself must be installed on your system. Most Linux systems provide
 Python through the native package manager. For Windows users the most straightforward approach for installing
 Python is through one of the prepackaged Python distributions such as `Enthought Canopy`_,
@@ -20,14 +19,20 @@ The dependencies of PCSE are the following:
 * Numpy >= 1.6
 * SQLalchemy >= 0.8
 * PyYAML >= 3.11
-* tabulate >= 0.7.5
 * xlrd >= 0.9.3
 * xlwt >= 1.0.0
+* pandas >= 0.20
+* requests >= 2.0.0
+* traitlets-pcse == 5.0.0.dev
+
+The last package in the list is a modified version of the `traitlets`_ package which provides some
+additional functionality used by PCSE.
 
 .. _Enthought Canopy: https://www.enthought.com/products/canopy/
 .. _Anaconda: https://store.continuum.io/cshop/anaconda/
 .. _PythonXY: https://python-xy.github.io/
 .. _HomeBrew: http://brew.sh
+.. _traitlets: https://traitlets.readthedocs.io/en/stable/
 
 Setting up your python environment
 ==================================
@@ -72,75 +77,46 @@ After installing MiniConda you should open a command box and check that conda is
         is foreign system : False
 
 Now we will use a Conda environment file to recreate the python environment that we use to develop and run
-PCSE. First you should download the conda environment file which comes in two flavours, a bare minimum
-environment for running PCSE (:download:`downloads/py2_pcse.yml`) and a more complete environment which
-includes the Jupyter notebook, Pandas and IPython (:download:`downloads/py2_pcse_full.yml`). If you intend
-to run the 'Getting started' section, you should take the latter one. Save the environment file
+PCSE. First you should download the conda environment file which comes in two flavours, an
+environment for running PCSE  on python 3 (:download:`downloads/py3_pcse.yml`) and one for python 2
+(:download:`downloads/py3_pcse.yml`). Both environments include the Jupyter notebook and IPython which are
+needed for running the `getting started` section and the example notebooks. Save the environment file
 on a temporary location such as ``d:\temp\make_env\``. We will now create a dedicated virtual environment
-using the command ``conda env create`` and tell conda to use the environment file with the option ``-f p2_pcse_full.yml``
+using the command ``conda env create`` and tell conda to use the environment file with the option ``-f p3_pcse.yml``
 as show below:
 
 .. code-block:: doscon
 
-    D:\temp\make_env>conda env create -f py2_pcse_full.yml
-    Fetching package metadata: ....
-    .Solving package specifications: .........
-    Linking packages ...
-    [      COMPLETE      ]|##################################################| 100%
-    Requirement already satisfied (use --upgrade to upgrade): backports.ssl-match-hostname==3.4.0.2 in c:\miniconda3\envs\py2_pcse_b\lib\site-packages
-    Requirement already satisfied (use --upgrade to upgrade): line-profiler==1.0 in c:\miniconda3\envs\py2_pcse_b\lib\site-packages
-    Requirement already satisfied (use --upgrade to upgrade): sphinx-rtd-theme==0.1.7 in c:\miniconda3\envs\py2_pcse_b\lib\site-packages
-    Collecting tabulate==0.7.5
-    Installing collected packages: tabulate
-    Successfully installed tabulate-0.7.5
+    (C:\Miniconda3) D:\temp\make_env>conda env create -f py3_pcse.yml
+    Fetching package metadata .............
+    Solving package specifications: .
+    intel-openmp-2 100% |###############################| Time: 0:00:00   6.39 MB/s
+
+    ... Lots of output here
+
+    Installing collected packages: traitlets-pcse
+    Successfully installed traitlets-pcse-5.0.0.dev0
     #
     # To activate this environment, use:
-    # > activate py2_pcse
+    # > activate py3_pcse
+    #
+    # To deactivate an active environment, use:
+    # > deactivate
+    #
+    # * for power-users using bash, you must source
     #
 
-You can then activate your environment (note the addition of ``[py2_pcse]`` on your command prompt) and
+You can then activate your environment (note the addition of ``(py3_pcse)`` on your command prompt) and
 run an upgrade to upgrade all packages to the latest versions:
 
 .. code-block:: doscon
 
-    D:\temp\make_env>activate py2_pcse
+    D:\temp\make_env>activate py3_pcse
     Deactivating environment "C:\Miniconda3"...
     Activating environment "C:\Miniconda3\envs\py2_pcse"...
 
-    [py2_pcse] D:\temp\make_env>conda upgrade --all
-    Fetching package metadata: ....
-    Solving package specifications: .........
-
-    Package plan for installation in environment C:\Miniconda3\envs\py2_pcse:
-
-    The following packages will be downloaded:
-
-        package                    |            build
-        ---------------------------|-----------------
-        vs2008_runtime-9.00.30729.1|                1         1.1 MB
-        zlib-1.2.8                 |            vc9_3          95 KB
-
-    ... Lots of output here
-
-    Proceed ([y]/n)? y
-
-    Fetching packages ...
-    vs2008_runtime 100% |###############################| Time: 0:00:02 518.43 kB/s
-
-    ... Lots of output here
-
-    pyqt-4.11.4-py 100% |###############################| Time: 0:00:04 571.51 kB/s
-    tornado-4.3-py 100% |###############################| Time: 0:00:01 387.81 kB/s
-    Extracting packages ...
-    [      COMPLETE      ]|##################################################| 100%
-    Unlinking packages ...
-    [      COMPLETE      ]|##################################################| 100%
-    Linking packages ...
-            1 file(s) copied.######################################          |  81%
-    [      COMPLETE      ]|##################################################| 100%
-
-    [py2_pcse] D:\temp\make_env>
-
+    D:\temp\make_env>activate py3_pcse
+    (py3_pcse) D:\temp\make_env>
 
 Installing and testing PCSE
 ===========================
@@ -154,7 +130,7 @@ environment:
 
 .. code-block:: doscon
 
-    [py2_pcse] D:\temp\make_env>pip install PCSE
+    (py3_pcse) D:\temp\make_env>pip install pcse
     Collecting PCSE
     Requirement already satisfied (use --upgrade to upgrade): numpy>=1.6.0 in c:\miniconda3\envs\py2_pcse\lib\site-packages (from PCSE)
     Requirement already satisfied (use --upgrade to upgrade): xlrd>0.9.0 in c:\miniconda3\envs\py2_pcse\lib\site-packages (from PCSE)
@@ -189,20 +165,18 @@ For testing the PCSE package we need to start python and import pcse:
 
 .. code-block:: doscon
 
-    [py2_pcse] D:\temp\make_env>python
-    Python 2.7.11 |Continuum Analytics, Inc.| (default, Mar  4 2016, 15:18:41) [MSC v.1500 32 bit (Intel)] on win32
+    (py3_pcse) D:\temp\make_env>python
+    Python 3.7.0 (default, Aug 14 2018, 19:12:50) [MSC v.1900 32 bit (Intel)] :: Anaconda, Inc. on win32
     Type "help", "copyright", "credits" or "license" for more information.
-    Anaconda is brought to you by Continuum Analytics.
-    Please check out: http://continuum.io/thanks and https://anaconda.org
     >>> import pcse
-    Building PCSE demo database at: C:\Users\wit015\.pcse\pcse.db
+    Building PCSE demo database at: C:\Users\wit015\.pcse\pcse.db ... OK
     >>>
 
 Next, the tests can be executed by calling the `test()` function at the top of the package::
 
+
     >>> pcse.test()
     runTest (pcse.tests.test_abioticdamage.Test_FROSTOL) ... ok
-    runTest (pcse.tests.test_assimilation.Test_WOFOST_Assimilation) ... ok
     runTest (pcse.tests.test_partitioning.Test_DVS_Partitioning) ... ok
     runTest (pcse.tests.test_evapotranspiration.Test_PotentialEvapotranspiration) ... ok
     runTest (pcse.tests.test_evapotranspiration.Test_WaterLimitedEvapotranspiration1) ... ok
@@ -220,25 +194,26 @@ Next, the tests can be executed by calling the `test()` function at the top of t
     runTest (pcse.tests.test_agromanager.TestAgroManager6) ... ok
     runTest (pcse.tests.test_agromanager.TestAgroManager7) ... ok
     runTest (pcse.tests.test_agromanager.TestAgroManager8) ... ok
-    runTest (pcse.tests.test_wofost.TestPotentialSunflower) ... ok
     runTest (pcse.tests.test_wofost.TestWaterlimitedWinterWheat) ... ok
+    runTest (pcse.tests.test_wofost.TestWaterlimitedWinterRapeseed) ... ok
     runTest (pcse.tests.test_wofost.TestWaterlimitedSunflower) ... ok
+    runTest (pcse.tests.test_wofost.TestPotentialWinterRapeseed) ... ok
+    runTest (pcse.tests.test_wofost.TestPotentialSpringBarley) ... ok
+    runTest (pcse.tests.test_wofost.TestPotentialSunflower) ... ok
     runTest (pcse.tests.test_wofost.TestPotentialPotato) ... ok
+    runTest (pcse.tests.test_wofost.TestWaterlimitedGrainMaize) ... ok
     runTest (pcse.tests.test_wofost.TestPotentialWinterWheat) ... ok
+    runTest (pcse.tests.test_wofost.TestPotentialGrainMaize) ... ok
     runTest (pcse.tests.test_wofost.TestWaterlimitedSpringBarley) ... ok
-    runTest (pcse.tests.test_wofost.TestWaterlimitedGrainMaize) ... ok
-    runTest (pcse.tests.test_wofost.TestWaterlimitedWinterRapeseed) ... ok
-    runTest (pcse.tests.test_wofost.TestPotentialWinterRapeseed) ... ok
     runTest (pcse.tests.test_wofost.TestWaterlimitedPotato) ... ok
-    runTest (pcse.tests.test_wofost.TestPotentialSpringBarley) ... ok
-    runTest (pcse.tests.test_wofost.TestPotentialGrainMaize) ... ok
     runTest (pcse.tests.test_lintul3.TestLINTUL3_SpringWheat) ... ok
     runTest (pcse.tests.test_wofost_npk.TestWOFOSTNPK_WinterWheat) ... ok
 
     ----------------------------------------------------------------------
-    Ran 33 tests in 57.472s
+    Ran 32 tests in 54.306s
 
     OK
+    >>>
 
 If the model output matches the expected output the test will report 'OK',
 otherwise an error will be produced with a detailed traceback on where the
@@ -248,10 +223,8 @@ due to problems with the test.
 
 Moreover, SQLAlchemy may complain with a warning that can be safely ignored::
 
-     /usr/lib/python2.7/dist-packages/sqlalchemy/types.py:307: SAWarning:
-     Dialect sqlite+pysqlite does *not* support Decimal objects natively, and
-     SQLAlchemy must convert from floating point - rounding errors and other
-     issues may occur. Please consider storing Decimal numbers as strings or
-     integers on this platform for lossless storage.
-         d[coltype] = rp = d['impl'].result_processor(dialect, coltype)
+    C:\Miniconda3\envs\py3_pcse\lib\site-packages\sqlalchemy\sql\sqltypes.py:603: SAWarning:
+    Dialect sqlite+pysqlite does *not* support Decimal objects natively, and SQLAlchemy must
+    convert from floating point - rounding errors and other issues may occur. Please consider
+    storing Decimal numbers as strings or integers on this platform for lossless storage.
 

doc/installing.rst

@@ -1,3 +1,13 @@
+What's new in PCSE 5.4
+======================
+
+PCSE 5.4 has the following new features:
+
+- PCSE is now fully compatible with python3 (>3.4) while still remaining compatibility with python 2.7.14
+- The NASAPOWERWeatherDataProvider has been upgraded to take the new API into account
+
+
+
 What's new in PCSE 5.3
 ======================