add feature to share SqlContext

[huozhanfeng]

I want to preprocessing some data and create tables in a sqlContext and all question later can share this sqlContext object and process data base on the sqlContext, just as the effect of namedRdds.

I feel this is a popular demand and it can help others a lot.

Thanks.

[velvia]

Absolutely. Understand this is a feature request. It's in the plans, but
personally I probably won't have time to work on it for another month.

-Evan

On Tue, Sep 23, 2014 at 12:24 AM, Zhanfeng Huo [email protected]
wrote:

I want to preprocessing some data and create tables in a sqlContext and
all question later can share this sqlContext object and processing data
base on the sqlContext, just as the effect of namedRdds.

I feel this is a popular demand and it can help others a lot.

Thanks.

—
Reply to this email directly or view it on GitHub
#65.

The fruit of silence is prayer;
the fruit of prayer is faith;
the fruit of faith is love;
the fruit of love is service;
the fruit of service is peace. -- Mother Teresa

[huozhanfeng]

I have implements a solution for internal use by extending the interface with spark 1.1.0. But I don't know whether it is a reasonable design for it forces into a parameter that may not be used.

@def runJob(sc: SparkContext, sqlContext: SQLContext, jobConfig: Config): Any

Another idea is realize a feature like NamedSqlContext, I feel this will be better.

[velvia]

Hi Zhanfeng,

Ni hao.
I think your solution, well, the issue is that it only works for
SQLContext. I'm aiming to make a generic solution that would work for just
about any derivative of SparkContext or related contexts. How it works
exactly I'm not sure, I'll have to think about it a little more.

The NamedRdds stuff may need to change a little bit, but maybe not, all it
really needs is a way to tie into the SparkContext underlying the
SQLContext. Or you can just rely on the table() method to discover your
tables. :)

-Evan

On Tue, Sep 23, 2014 at 4:38 AM, Zhanfeng Huo [email protected]
wrote:

I have implements a solution for internal use by extending the interface
with spark 1.1.0. But I don't know whether it is a reasonable design for it
forced into a parameter that may not be used.

@def https://github.com/def runJob(sc: SparkContext, sqlContext:
SQLContext, jobConfig: Config): Any

Another idea is realize a feature like NamedSqlContext, I fell this will
be better.

—
Reply to this email directly or view it on GitHub
#65 (comment)
.

The fruit of silence is prayer;
the fruit of prayer is faith;
the fruit of faith is love;
the fruit of love is service;
the fruit of service is peace. -- Mother Teresa

[huozhanfeng]

That's great. I'm interested in it and I hope to make a contribution to this project to help others if time permits.

[velvia]

I'll share a design on the mailing list when the time comes closer....
sounds good?

On Tue, Sep 23, 2014 at 11:02 PM, Zhanfeng Huo [email protected]
wrote:

That's great. I'm interested in it and I hope to make a contribution to
this project to help others if time permits.

—
Reply to this email directly or view it on GitHub
#65 (comment)
.

The fruit of silence is prayer;
the fruit of prayer is faith;
the fruit of faith is love;
the fruit of love is service;
the fruit of service is peace. -- Mother Teresa

[huozhanfeng]

That's good. I will keep attention to it.

[sranga]

This feature is something that I am looking forward to as well.

[nightwolfzor]

+1 - will this also support HiveContext?