diff --git a/CHANGES.txt b/CHANGES.txt index 99cd0bbbf..6b6e3244d 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,5 +1,11 @@ +3.2.0 + * Spark 3.2.x support (SPARKC-670) + * Fix: Cassandra Direct Join doesn't quote keyspace and table names (SPARKC-667) + * Fix: The connector can't find a codec for BLOB <-> java.nio.HeapByteBuffer (SPARKC-673) + 3.1.1 * Fix: Cassandra Direct Join doesn't quote keyspace and table names (SPARKC-667) + * Fix: The connector can't find a codec for BLOB <-> java.nio.HeapByteBuffer (SPARKC-673) 3.1.0 * Updated Spark to 3.1.1 and commons-lang to 3.10 (SPARKC-626, SPARKC-646) @@ -8,6 +14,7 @@ 3.0.2 * Fix: Cassandra Direct Join doesn't quote keyspace and table names (SPARKC-667) + * Fix: The connector can't find a codec for BLOB <-> java.nio.HeapByteBuffer (SPARKC-673) 3.0.1 * Fix: repeated metadata refresh with the Spark connector (SPARKC-633) diff --git a/README.md b/README.md index 2dc4d06ec..4cd052bed 100644 --- a/README.md +++ b/README.md @@ -4,11 +4,11 @@ ## Quick Links -| What | Where | -| ---------- | ----- | -| Community | Chat with us at [Datastax and Cassandra Q&A](https://community.datastax.com/index.html) | -| Scala Docs | Most Recent Release (3.1.0): [Spark-Cassandra-Connector](https://datastax.github.io/spark-cassandra-connector/ApiDocs/3.1.0/connector/com/datastax/spark/connector/index.html), [Spark-Cassandra-Connector-Driver](https://datastax.github.io/spark-cassandra-connector/ApiDocs/3.1.0/driver/com/datastax/spark/connector/index.html)| -| Latest Production Release | [3.1.0](https://search.maven.org/artifact/com.datastax.spark/spark-cassandra-connector_2.12/3.1.0/jar) | +| What | Where | +| ---------- |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Community | Chat with us at [Datastax and Cassandra Q&A](https://community.datastax.com/index.html) | +| Scala Docs | Most Recent Release (3.2.0): [Spark-Cassandra-Connector](https://datastax.github.io/spark-cassandra-connector/ApiDocs/3.2.0/connector/com/datastax/spark/connector/index.html), [Spark-Cassandra-Connector-Driver](https://datastax.github.io/spark-cassandra-connector/ApiDocs/3.2.0/driver/com/datastax/spark/connector/index.html) | +| Latest Production Release | [3.2.0](https://search.maven.org/artifact/com.datastax.spark/spark-cassandra-connector_2.12/3.2.0/jar) | ## Features @@ -19,7 +19,7 @@ Spark RDDs and Datasets/DataFrames to Cassandra tables, and execute arbitrary CQ in your Spark applications. - Compatible with Apache Cassandra version 2.1 or higher (see table below) - - Compatible with Apache Spark 1.0 through 3.1 ([see table below](#version-compatibility)) + - Compatible with Apache Spark 1.0 through 3.2 ([see table below](#version-compatibility)) - Compatible with Scala 2.11 and 2.12 - Exposes Cassandra tables as Spark RDDs and Datasets/DataFrames - Maps table rows to CassandraRow objects or tuples @@ -44,9 +44,11 @@ named "bX.Y" where X.Y is the major+minor version; for example the "b1.6" branch corresponds to the 1.6 release. The "master" branch will normally contain development for the next connector release in progress. -Currently the following branches are actively supported: 3.1.x ([master](https://github.com/datastax/spark-cassandra-connector/tree/master)), -3.0.x ([b3.0](https://github.com/datastax/spark-cassandra-connector/tree/b3.0)) and 2.5.x -([b2.5](https://github.com/datastax/spark-cassandra-connector/tree/b2.5)). +Currently, the following branches are actively supported: +3.2.x ([master](https://github.com/datastax/spark-cassandra-connector/tree/master)), +3.1.x ([b3.1](https://github.com/datastax/spark-cassandra-connector/tree/b3.1)), +3.0.x ([b3.0](https://github.com/datastax/spark-cassandra-connector/tree/b3.0)) and +2.5.x ([b2.5](https://github.com/datastax/spark-cassandra-connector/tree/b2.5)). | Connector | Spark | Cassandra | Cassandra Java Driver | Minimum Java Version | Supported Scala Versions | | --------- | ------------- | --------- | --------------------- | -------------------- | ----------------------- | @@ -71,6 +73,9 @@ Currently the following branches are actively supported: 3.1.x ([master](https:/ ## Hosted API Docs API documentation for the Scala and Java interfaces are available online: +### 3.2.0 +* [Spark-Cassandra-Connector](https://datastax.github.io/spark-cassandra-connector/ApiDocs/3.2.0/connector/com/datastax/spark/connector/index.html) + ### 3.1.0 * [Spark-Cassandra-Connector](https://datastax.github.io/spark-cassandra-connector/ApiDocs/3.1.0/connector/com/datastax/spark/connector/index.html) @@ -90,7 +95,7 @@ This project is available on the Maven Central Repository. For SBT to download the connector binaries, sources and javadoc, put this in your project SBT config: - libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" % "3.1.0" + libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" % "3.2.0" * The default Scala version for Spark 3.0+ is 2.12 please choose the appropriate build. See the [FAQ](doc/FAQ.md) for more information. diff --git a/doc/0_quick_start.md b/doc/0_quick_start.md index 9d9a82e33..7a3b8fdca 100644 --- a/doc/0_quick_start.md +++ b/doc/0_quick_start.md @@ -15,14 +15,14 @@ Configure a new Scala project with the Apache Spark and dependency. The dependencies are easily retrieved via Maven Central - libraryDependencies += "com.datastax.spark" % "spark-cassandra-connector_2.12" % "3.1.0" + libraryDependencies += "com.datastax.spark" % "spark-cassandra-connector_2.12" % "3.2.0" The spark-packages libraries can also be used with spark-submit and spark shell, these commands will place the connector and all of its dependencies on the path of the Spark Driver and all Spark Executors. - $SPARK_HOME/bin/spark-shell --packages com.datastax.spark:spark-cassandra-connector_2.12:3.1.0 - $SPARK_HOME/bin/spark-submit --packages com.datastax.spark:spark-cassandra-connector_2.12:3.1.0 + $SPARK_HOME/bin/spark-shell --packages com.datastax.spark:spark-cassandra-connector_2.12:3.2.0 + $SPARK_HOME/bin/spark-submit --packages com.datastax.spark:spark-cassandra-connector_2.12:3.2.0 For the list of available versions, see: - https://repo1.maven.org/maven2/com/datastax/spark/spark-cassandra-connector_2.12/ @@ -42,7 +42,7 @@ and *all* of its dependencies on the Spark Class PathTo configure the default Spark Configuration pass key value pairs with `--conf` $SPARK_HOME/bin/spark-shell --conf spark.cassandra.connection.host=127.0.0.1 \ - --packages com.datastax.spark:spark-cassandra-connector_2.12:3.1.0 + --packages com.datastax.spark:spark-cassandra-connector_2.12:3.2.0 --conf spark.sql.extensions=com.datastax.spark.connector.CassandraSparkExtensions This command would set the Spark Cassandra Connector parameter diff --git a/doc/13_spark_shell.md b/doc/13_spark_shell.md index ee6eb3964..9a704aff8 100644 --- a/doc/13_spark_shell.md +++ b/doc/13_spark_shell.md @@ -18,7 +18,7 @@ Find additional versions at [Spark Packages](https://repo1.maven.org/maven2/com/ ```bash cd spark/install/dir #Include the --master if you want to run against a spark cluster and not local mode -./bin/spark-shell [--master sparkMasterAddress] --jars yourAssemblyJar --packages com.datastax.spark:spark-cassandra-connector_2.12:3.1.0 --conf spark.cassandra.connection.host=yourCassandraClusterIp +./bin/spark-shell [--master sparkMasterAddress] --jars yourAssemblyJar --packages com.datastax.spark:spark-cassandra-connector_2.12:3.2.0 --conf spark.cassandra.connection.host=yourCassandraClusterIp ``` By default spark will log everything to the console and this may be a bit of an overload. To change this copy and modify the `log4j.properties` template file diff --git a/doc/15_python.md b/doc/15_python.md index 47a407337..4b791314a 100644 --- a/doc/15_python.md +++ b/doc/15_python.md @@ -14,7 +14,7 @@ shell similarly to how the spark shell is started. The preferred method is now t ```bash ./bin/pyspark \ - --packages com.datastax.spark:spark-cassandra-connector_2.12:3.1.0 \ + --packages com.datastax.spark:spark-cassandra-connector_2.12:3.2.0 \ --conf spark.sql.extensions=com.datastax.spark.connector.CassandraSparkExtensions ``` diff --git a/doc/developers.md b/doc/developers.md index 62551c0e9..c271ede57 100644 --- a/doc/developers.md +++ b/doc/developers.md @@ -31,11 +31,11 @@ Cassandra and Spark nodes and are the core of our test coverage. ### Merge Path -b2.5 => b3.0 => master +b2.5 => b3.0 => b3.1 => master New features can be considered for 2.5 as long as they do not break apis. Once a feature is ready for b2.5, create a feature branch for b3.0 and merge -b2.5 feature branch to b3.0 feature branch. Repeat for master. +b2.5 feature branch to b3.0 feature branch. Repeat for b3.1 and master. Example for imaginary SPARKC-9999. @@ -62,10 +62,17 @@ git merge SPARKC-9999-b2.5 # Resolve conflict, if any # Push the new feature branch: git push origin SPARKC-9999-b3.0 + +# Forward merge on the next version: +git checkout -b SPARKC-9999-b3.1 datastax/b3.1 +git merge SPARKC-9999-b3.0 +# Resolve conflict, if any +# Push the new feature branch: +git push origin SPARKC-9999-b3.1 # Forward merge on the next version: git checkout -b SPARKC-9999-master datastax/master -git merge SPARKC-9999-b3.0 +git merge SPARKC-9999-b3.1 # Resolve conflict, if any # Push the new feature branch: git push origin SPARKC-9999-master