Replace networkx with rustworkx #9504
thomasaarholt
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Large (1000+ models) dbt projects can take quite a long time to build.
Part of that time is assumedly spent constructing a complex DAG of all the models in the project.
I see that networkx is used in dbt, assumedly for this type of graph construction? If a significant amount of time is spent on this part, then it might be worth considering switching to the "spiritual successor" of networkx, rustworkx. It is significantly faster, as shown in these benchmarks: https://www.rustworkx.org/benchmarks.html
It isn't a drop-in replacement, but it is not too far from it.
Beta Was this translation helpful? Give feedback.
All reactions