Update for Version 8 export

Added thread lock around register function to avoid issues with Scala 10.x reflection.
Added the ability to specify the Spark persistence mode for generated RDD and add Kyro registration.

Patch for Version 7 missing SolarGeneratingUnit locations

Fixes an exception reading in SolarGeneratingUnit elements without locations.

Reading in the full export files leads to an exception (six times):

java.util.NoSuchElementException: key not found: location
at scala.collection.MapLike$class.default(MapLike.scala:228)
at scala.collection.AbstractMap.default(Map.scala:58)
at scala.collection.mutable.HashMap.apply(HashMap.scala:64)
at ch.ninecode.SolarGeneratingUnit$.unpickle(CIM.scala:1387)
at ch.ninecode.CIM.parse_one(CIM.scala:1802)
at ch.ninecode.CIMRecordReader.nextKeyValue(CIMRecordReader.scala:77)

The exception location (at ch.ninecode.SolarGeneratingUnit$.unpickle(CIM.scala:1387)) indicates the CIMScala-1.4.1-SNAPSHOT.jar version used was 0.4.1 and not the current version 0.5.0.

Despite that, the error would have occurred in the new version anyway, the problem being an absence of PowerSystemResource.Location attribute in the SolarGeneratingUnit element.

This may be due to new additions (since February) now exported in the data from August NIS_CIM_Export_sias_current_20160816_V7.

The mandatory requirement for a location was removed from the SolarGeneratingUnit element.

Patch for NIS export V7

This fix tries a workaround for a threading issue when registering the CHIM classes (and hence does not apply to the Spark 1.4.1 branch).

Theory

Sometimes when registering the many RDDs of case classes as tables we run into a Scala 2.10 thread safety issue (described by Michael Armbrust who closed https://issues.apache.org/jira/browse/SPARK-2178 because it depends on https://issues.scala-lang.org/browse/SI-6240 - which will never be fixed.

The following are examples of the strange things that can happen - although not consistently:

• scala.reflect.runtime.ReflectError: value ch is not a package
• java.lang.RuntimeException: error reading Scala signature of ch.ninecode.model.BusBarSectionInfo: value model is not a package
• java.lang.NullPointerException

with this in the stack trace somewhere:

 at ch.ninecode.model.Parseable.register(CHIM.scala:179)
 at ch.ninecode.model._AssetInfo$.register(CHIM.scala:1396)
 at ch.ninecode.model.CHIM.<init>(CHIM.scala:6279)

Update for Version 7 export

The NIS CIM Version 7 export has many changes, of which the following affect the CIMScala reader:

• add Jumper to handle Trenmesser (a metal jumper in a fuse holder used to connect/disconnect a feeder)
• add Conductor for reference connections (virtual connections between overhead and underground)
• PowerTransformerEnd replaces the more complex modelling with TransformerTank and TransformerTankEnd

Update for Version 7 export

The NIS CIM Version 7 export has many changes, of which the following affect the CIMScala reader:

• add Jumper to handle Trenmesser (a metal jumper in a fuse holder used to connect/disconnect a feeder)
• add Conductor for reference connections (virtual connections between overhead and underground)
• PowerTransformerEnd replaces the more complex modelling with TransformerTank and TransformerTankEnd

Not all of the attribute changes were incorporated in this Spark version 1.4.1 branch, since that will be retired in the August/September 2016 time frame and users should migrate to the master branch (Spark version 1.6.0) as soon as possible.

Patch for yarn cluster mode test

Changed the creation of SparkContext in Sample.scala, not to specify client mode, allowing it to run in cluster mode (via the command line):
bin/spark-submit --verbose --class ch.ninecode.Sample --master yarn --deploy-mode cluster etc.

Update for NIS export V6a

Adds SvStatus to edge table.
Also fully implements hierarchical classes.

Update for NIS export V6a

Adds SvStatus to edges table.

Update for NIS export V4 patch #2

Fix SolarGeneratingUnit.commissioned.

Update for NIS export V4 patch

Fix Location loss.