Skip to content

Commit

Permalink
Updated readme file
Browse files Browse the repository at this point in the history
  • Loading branch information
adityajaroli committed Jan 11, 2024
1 parent b5afa19 commit 217fafb
Show file tree
Hide file tree
Showing 4 changed files with 25 additions and 2 deletions.
23 changes: 23 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,27 @@ This utility leverages the power of PostgreSQL in combination with Python to eff
4. Harnessing the power of multiprocessing
5. Capability to drop indexes during insertion and recreate them in parallel

<h2>package's Efficiency</h2>

**Machine:**
- Resource config - 5 core, 8GB
- Azure hosted PostgreSQL Server
- Azure hosted Python service (jupyter notebook)

**Table info:**
- 12 columns (3 texts, 2 date, 7 double)
- Primary key: 3 columns (2 text and 1 date)
- Indexes: 2 b-tree. (1 on single column and another on three columns)

**Runtime:**
- Data Size: 20M
- without PK and Indexes: ~55s
- with PK and indexes: ~150s (~85s to insert data with PK enabled and ~65 seconds to create indexes)

**With the same above data setup, running it on locally hosted PostgreSQL DB:**

![Screenshot](localruntime.png)

<h2>Usage</h2>

The utility provides the following useful functions and classes:
Expand Down Expand Up @@ -189,3 +210,5 @@ async def run():
if __name__ == '__main__':
asyncio.run(run())
```


Binary file added localruntime.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ classifiers = [

dependencies = [
"pandas",
"psycopg[binary]",
"psycopg",
"asyncio",
"psycopg_pool",
"retry"
Expand Down
2 changes: 1 addition & 1 deletion requirements.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
pandas
psycopg[binary]
psycopg
asyncio
psycopg_pool
retry

0 comments on commit 217fafb

Please sign in to comment.