Merge pull request #58 from lanl-ansi/pm-data-rev

Reworking Data Processing

Author: rb004f

.codecov.yml

@@ -0,0 +1,7 @@
+status:
+    project:                   # measuring the overall project coverage
+      default:                 # context, you can create multiple ones with custom titles
+        enabled: yes           # must be yes|true to enable this status
+        target: 80%
+      branches:              # -> see "branch patterns" below
+        threshold: 10%        # allowed to drop X% and still result in a "success" commit status

CHANGELOG.md

@@ -1,8 +1,13 @@
 PowerModels.jl Change Log 
 =================
 
-### staged
+### Staged
 - Updated to JuMP v0.15 syntax
+- Replaced PowerModels set data types with "ref" dictionary
+- Refactored Matpower data processing to simplify editing data after parsing
+- Replaced JSON test files with Matpower test files
+- Added documentation on internal JSON data format to DATA.md
+- Updated TNEP models to work with Matpower parsing extensions
 
 ### v0.2.3
 - Multiple improvements to Matlab file parsing

DATA.md

@@ -1,9 +1,9 @@
 # PowerModels Data Format
 
-## Network Data Dictionary
+## The Network Data Dictionary
 
 Internally PowerModels utilizes a dictionary to store network data.
-The dictionary uses strings as key values so this data structure can be serialized to JSON for algorithmic data exchange.
+The dictionary uses strings as key values so it can be serialized to JSON for algorithmic data exchange.
 The data dictionary organization and key names are designed to be consistent with the [Matpower](http://www.pserc.cornell.edu/matpower/) file format and should be familiar to power system researchers.
 
 The network data dictionary structure is roughly as follows,
@@ -41,26 +41,32 @@ The network data dictionary structure is roughly as follows,
         ...
     },
     ...
-],
-"gencost":[
-    {
-        "index":<int>,
-        "model":<int>,
-        "startup":<float>,
-        "shutdown":<float>,
-        ...
-    },
-    ...
 ]
 }
 ```
 
-For a detailed list of all possible parameters please refer to the specification document provided with [Matpower](http://www.pserc.cornell.edu/matpower/).  In addition to the traditional Matpower parameters every network component in the PowerModels dictionary has an `index` parameter, which can be used to uniquely identify that network element.
+The following commands can be used to explore the network data dictionary generated by a given Matpower data file,
+```
+network_data = PowerModels.parse_file("$(Pkg.dir("PowerModels"))/test/data/case14.m")
+display(network_data)
+```
+
+For a detailed list of all possible parameters refer to the specification document provided with [Matpower](http://www.pserc.cornell.edu/matpower/).  
+
+### Noteworthy Differences from Matpower Data Files
+
+The PowerModels network data dictionary differs from the Matpower format in the following ways,
 
-It is also important to note that although the Matpower format contains values in mixed units, during PowerModels data setup phase all of the data values are converted to per-unit and radian values.
+- All PowerModels components have an `index` parameter, which can be used to uniquely identify that network element.
+- All network parameters are in per-unit and angles are in radians.
+- All non-transformer branches are given nominal transformer values (i.e. a tap of 1.0 and a shift of 0).
+- When present, the `gencost` data is incorporated into the `gen` data, the column names remain the same.
+- Only quadratic active power generation cost functions are supported at this time.
+- Some additional derived values are added to branches, such as line admittance values (see `update_derived_values` in `src/core/data.jl` for details).
+- Special treatment is given to the optional `ne_branch` matrix to support the TNEP problem.
 
 
-## Matpower Data Files
+## Working with Matpower Data Files
 
 The data exchange via JSON files is ideal for building algorithms, however it is hard to for humans to read and process.
 To that end PowerModels also has extensive support for parsing Matpower network files in the `.m` format.
@@ -135,7 +141,7 @@ becomes,
 }
 ```
 
-#### Standard Matrix Extentions
+#### Standard Matrix Extensions
 Finally, if a nonstandard matrix's name extends a current Matpower matrix name with an underscore, then its values will be merged with the original Matpower component data.  Note that this feature requires that the nonstandard matrix has column names and has the same number of rows as the original matrix (similar to the `gencost` matrix in the Matpower format).  For example,
 ```
 %column_names%  rate_i  rate_p

README.md

@@ -37,7 +37,7 @@ For the current development version, "checkout" this package with,
 
 `Pkg.checkout("PowerModels")`
 
-At least one solver is required for running PowerModels.  The open-source solver Ipopt is recommended, as it is extremely fast, and can be used to solve a wide variety of the problems and network formulations provided in PowerModels.  The Ipopt solver can be installed via tha package manager with,
+At least one solver is required for running PowerModels.  The open-source solver Ipopt is recommended, as it is extremely fast, and can be used to solve a wide variety of the problems and network formulations provided in PowerModels.  The Ipopt solver can be installed via the package manager with,
 
 `Pkg.add("Ipopt")`
 
@@ -71,6 +71,23 @@ run_opf("nesta_case3_lmbd.m", SOCWRPowerModel, IpoptSolver())
 Extending PowerModels with new problems and formulations will be covered in a another tutorial, that is not yet available.
 
 
+### Modifying Network Data
+
+The follow example demonstrates how to perform multiple PowerModels solves while modify the network data in Julia,
+
+```
+network_data = PowerModels.parse_file("nesta_case3_lmbd.m")
+run_opf(network_data, ACPPowerModel, IpoptSolver())
+
+network_data["bus"][3]["pd"] = 0.0
+network_data["bus"][3]["qd"] = 0.0
+
+run_opf(network_data, ACPPowerModel, IpoptSolver())
+```
+
+For additional details about the PowerModels network data structure see the DATA.md file.
+
+
 ## Comparison to Other Tools
 
 Forthcoming.

src/PowerModels.jl

@@ -12,6 +12,7 @@ include("io/json.jl")
 include("io/common.jl")
 
 include("core/base.jl")
+include("core/data.jl")
 include("core/variable.jl")
 include("core/constraint.jl")
 include("core/relaxation_scheme.jl")