How to initialize sparkContext class in C#

[SatishkumarJha]

Hi ,

Can you plz. someone send any sample to run simple word count in C# with Spark ,I have tried but getting error

No connection could be made because the target machine actively refused it

on declaration of SparkContext var sparkContext = new SparkContext()

It seems configuration issue ,Can you plz. suggest how to pass spark running instance , my running instance is http://localhost:4040

Thanks.
Satish Jha

[jayprajapati857]

How you are trying to execute? From command line or from visual studio?

[SatishkumarJha]

from visual studio,I have installed Spark & tested sample code on command prompt which is working fine.
I have to use parallelism in C# with Spark.That's why I have installed Spark from nuget packages.
Sample code:
`using Microsoft.Spark.CSharp.Core;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace ConsoleApp5
{
class Program
{
public static void Main(string[] args)
{
string home = "http://127.0.0.1:4040";// "spark://hostname:port");
var sparkContext = new SparkContext(new SparkConf().SetAppName("test").SetMaster("Master").SetSparkHome(home));//error throwing here

        try
        {
            const int slices = 3;
            var numberOfItems = (int)Math.Min(100000L * slices, int.MaxValue);
            var values = new List<int>(numberOfItems);
            for (var i = 0; i <= numberOfItems; i++)
            {
                values.Add(i);
            }

            var rdd = sparkContext.Parallelize(values, slices);

            CalculatePiUsingAnonymousMethod(numberOfItems, rdd);

        }
        catch (Exception ex)
        {
        }

        sparkContext.Stop();

    }
    private static void CalculatePiUsingAnonymousMethod(int n, RDD<int> rdd)
    {
        var count = rdd
                        .Map(i =>
                        {
                            var random = new Random();
                            var x = random.NextDouble() * 2 - 1;
                            var y = random.NextDouble() * 2 - 1;

                            return (x * x + y * y) < 1 ? 1 : 0;
                        })
                        .Reduce((x, y) => x + y);

    }
}

}
`
Error:
System.Net.Sockets.SocketException occurred
HResult=0x80004005
Message=No connection could be made because the target machine actively refused it
Source=System
StackTrace:
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
at Microsoft.Spark.CSharp.Network.DefaultSocketWrapper.Connect(IPAddress remoteaddr, Int32 port)
at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.GetConnection()
at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] parameters)
at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallConstructor(String className, Object[] parameters)
at Microsoft.Spark.CSharp.Proxy.Ipc.SparkCLRIpcProxy.CreateSparkConf(Boolean loadDefaults)
at Microsoft.Spark.CSharp.Core.SparkConf..ctor(Boolean loadDefaults)
at ConsoleApp5.Program.Main(String[] args) in C:\Users\satish.jha\documents\visual studio 2017\Projects\ConsoleApp5\ConsoleApp5\Program.cs:line 15

[jayprajapati857]

If you are trying to execute it from visual studio then you need to try in debug mode to initialise the spark context have you read this link : https://github.com/Microsoft/Mobius/blob/master/notes/running-mobius-app.md#running-mobius-examples-in-local-mode

Just read and set the environment accordingly you will able to execute

[jayprajapati857]

Read the documentation for debug mode from that link

[SatishkumarJha]

Hi jayprajapati,

I tried to run this command 'sparkclr-submit.cmd --exe SparkCLRSamples.exe C:\Users\satish.jha\Desktop\Mobius-master\csharp\Samples\Microsoft.Spark.CSharp\bin\Debug' on scala/spark command but getting error plz. suggest where I'm doing mistake.
I have pasted my spark console below..plz. have a look .

Microsoft Windows [Version 10.0.10240]
(c) 2015 Microsoft Corporation. All rights reserved.

C:\Users\satish.jha>cd c:\spark

c:\Spark>spark-shell
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/11/04 12:13:51 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/11/04 12:14:02 WARN General: Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/c:/Spark/bin/../jars/datanucleus-rdbms-3.2.9.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/jars/datanucleus-rdbms-3.2.9.jar."
17/11/04 12:14:02 WARN General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/c:/Spark/bin/../jars/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/jars/datanucleus-core-3.2.10.jar."
17/11/04 12:14:03 WARN General: Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/c:/Spark/bin/../jars/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/jars/datanucleus-api-jdo-3.2.6.jar."
17/11/04 12:14:12 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Spark context Web UI available at http://192.168.22.103:4040
Spark context available as 'sc' (master = local[*], app id = local-1509777835771).
Spark session available as 'spark'.
Welcome to
____ __
/ / ___ / /
\ / _ / _ `/ __/ '/
// .__/_,// //_\ version 2.2.0
//

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_144)
Type in expressions to have them evaluated.
Type :help for more information.

scala

sparkclr-submit.cmd --exe SparkCLRSamples.exe C:\Users\satish.jha\Desktop\Mobius-master\csharp\Samples\Microsoft.Spark.CSharp\bin\Debug

:1: error: ';' expected but '.' found.
sparkclr-submit.cmd --exe SparkCLRSamples.exe C:\Users\satish.jha\Desktop\Mobius-master\csharp\Samples\Microsoft.Spark.CSharp\bin\Debug

                                     ^

Regards,
Satish