Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Python] Add support for df.show() on CREATE EXTERNAL TABLE statements #166

Open
andygrove opened this issue Aug 27, 2022 · 1 comment
Open
Labels
enhancement New feature or request

Comments

@andygrove
Copy link
Member

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

>>> df = ctx.sql("CREATE EXTERNAL TABLE orders STORED AS PARQUET LOCATION '/mnt/bigdata/tpch/sf1-parquet/orders'")
>>> df.show()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
Exception: Arrow error: External error: Execution error: Job hZh0A3x failed: Error planning job hZh0A3x: DataFusionError(Internal("Unsupported logical plan: CreateExternalTable"))

Describe the solution you'd like
Make this work

Describe alternatives you've considered
None

Additional context
None

@andygrove andygrove added the enhancement New feature or request label Aug 27, 2022
@andygrove
Copy link
Member Author

It actually does work but we get an error if we try and show the results of the DataFrame. This is not good UX so we will need to fix.

>>> import ballista
>>> ctx = ballista.BallistaContext("localhost", 50050)
>>> ctx.sql("CREATE EXTERNAL TABLE orders STORED AS PARQUET LOCATION '/mnt/bigdata/tpch/sf1-parquet/orders'")
<ballista.DataFrame object at 0x7f501f088150>
>>> df = ctx.sql("SELECT count(*) FROM orders")
>>> df.show()
+-----------------+
| COUNT(UInt8(1)) |
+-----------------+
| 14999999        |
+-----------------+

@andygrove andygrove changed the title Add support for CREATE EXTERNAL TABLE from Python [Python] Add support for df.show() on CREATE EXTERNAL TABLE statements Aug 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant