initial data connectors #82
Conversation
lotus/databases/connectors.py
Outdated
| class DatabaseConnector: | ||
| def __init__(self): | ||
| self.sql_engine = None | ||
| self.nosql_client = None |
There was a problem hiding this comment.
I think we should remove all the nosql stuff since there is not one interface for all the nosql offerings.
examples/db_examples/sql_db.py
Outdated
|
|
||
| connector = DatabaseConnector() | ||
| connector.connect_sql("sqlite:///example_movies.db") | ||
| lotus_db = LotusDB(connector) |
There was a problem hiding this comment.
Do we need LotusDB? Can't we just have a connect.load_as_pandas(table_name: str) directly?
There was a problem hiding this comment.
Maybe DatabaseConnector just provides a staticmethod
@staticmethod
def load_from_db(connection_url: str, table_name: str) -> pd.DataFrame
And then all the code here becomes
df = DatabaseConnector.load_from_db("sqlite:///example_movies.db", "movies")
There was a problem hiding this comment.
However, what if someone has a specific query thats not just select * from table? Maybe have something like
def load_from_db(connection_url: str, query: str) -> pd.DataFrame
There was a problem hiding this comment.
Also if someone has a very large dataset should we consider processing it in batches, and then concat them together?
There was a problem hiding this comment.
def load_from_db(connection_url: str, query: str) -> pd.DataFrame
This works
Also if someone has a very large dataset should we consider processing it in batches, and then concat them together?
If someone has a very large dataset then I think lotus just will not be able to work, since the requirement is that it can be loaded into a pandas dataframe in memory.
StanChan03
force-pushed
the
sc/sql_integration
branch
from
0048eb1 to
badfdd5
Compare
Initial SQL Integration