Update mermaid graph of onigumo processing with a new approach #237

nappex · 2024-08-03T18:42:40Z

No description provided.

nappex · 2024-08-03T18:43:20Z

You can play with graph by live service https://mermaid.live/

It is partially solving #232

Glutexo

Changing the diagram causes a slight mismatch in the description above it: although “Onigumo is composed of three sequentially interconnected components” is still correct in some sense, it doesn’t mention the new Feeder part. I’d say something like “The crawling part of Onigumo is composed of three sequentially interconnected components”. We can do that in a follow-up pull request.

It would be possible to object that the diagram doesn’t reflect the actual architecture of the project at the moment, but it didn’t do so before the change either: there is no Parser (will be resolved by #89), no Operator, no Materializer, no flow from start to end… So I think we don’t need to wait with merging for the actual feeder to emerge.

README.md

Glutexo

The diagrams are very nice! ❤️ I like how the Feeder, the starting point, sits almost in the upper-left corner and the Materialized, the finishing point, is located in the bottom-right corner, nicely flowing like a text on a page.

Glutexo · 2024-08-16T16:48:39Z

README.md

+        spider_parser(🕷️ PARSER)
+        spider_operator(🕷️ OPERATOR)


I like the idea of using the emoji to get around GitHub not supporting fontawesome icons in Mermaid diagrams. 💪🏻

Glutexo · 2024-08-16T17:13:58Z

README.md

+    start([START])  --> onigumo_feeder[FEEDER]
+    onigumo_feeder  -- .raw --> Crawling
+    onigumo_feeder  -- .urls --> Crawling
+    onigumo_feeder  -- .json --> Crawling
+    Crawling  --> spider_materializer(🕷️ MATERIALIZER)
+    spider_materializer --> done([END])


Can we only use a single space,

Suggested change

start([START]) --> onigumo_feeder[FEEDER]

onigumo_feeder -- .raw --> Crawling

onigumo_feeder -- .urls --> Crawling

onigumo_feeder -- .json --> Crawling

Crawling --> spider_materializer(🕷️ MATERIALIZER)

spider_materializer --> done([END])

start([START]) --> onigumo_feeder[FEEDER]

onigumo_feeder -- .raw --> Crawling

onigumo_feeder -- .urls --> Crawling

onigumo_feeder -- .json --> Crawling

Crawling --> spider_materializer(🕷️ MATERIALIZER)

spider_materializer --> done([END])

or align the items to table columns?

Suggested change

start([START]) --> onigumo_feeder[FEEDER]

onigumo_feeder -- .raw --> Crawling

onigumo_feeder -- .urls --> Crawling

onigumo_feeder -- .json --> Crawling

Crawling --> spider_materializer(🕷️ MATERIALIZER)

spider_materializer --> done([END])

start([START]) --> onigumo_feeder[FEEDER]

onigumo_feeder -- .raw --> Crawling

onigumo_feeder -- .urls --> Crawling

onigumo_feeder -- .json --> Crawling

Crawling --> spider_materializer(🕷️ MATERIALIZER)

spider_materializer --> done([END])

I don’t have preference (at least now), but I’d like it to be at least consistent.

Glutexo · 2024-08-16T17:14:38Z

README.md

+    spider_operator     -. "<hash>.urls" .-> onigumo_downloader
+    onigumo_downloader -. "<hash>.raw"  .->  spider_parser
+    spider_parser     -. "<hash>.json" .->  spider_operator


Suggested change

spider_operator -. "<hash>.urls" .-> onigumo_downloader

onigumo_downloader -. "<hash>.raw" .-> spider_parser

spider_parser -. "<hash>.json" .-> spider_operator

onigumo_downloader -. "<hash>.raw" .-> spider_parser

spider_operator -. "<hash>.urls" .-> onigumo_downloader

spider_parser -. "<hash>.json" .-> spider_operator

Suggested change

spider_operator -. "<hash>.urls" .-> onigumo_downloader

onigumo_downloader -. "<hash>.raw" .-> spider_parser

spider_parser -. "<hash>.json" .-> spider_operator

onigumo_downloader -. "<hash>.raw" .-> spider_parser

spider_operator. -. "<hash>.urls" .-> onigumo_downloader

spider_parser -. "<hash>.json" .-> spider_operator

Glutexo · 2024-08-16T17:17:25Z

README.md

+    onigumo_feeder  -- .json --> spider_operator
+    spider_operator ---> spider_materializer
+    onigumo_feeder  -- .urls --> onigumo_downloader
+    onigumo_feeder  -- .raw --> spider_parser


Suggested change

onigumo_feeder -- .json --> spider_operator

spider_operator ---> spider_materializer

onigumo_feeder -- .urls --> onigumo_downloader

onigumo_feeder -- .raw --> spider_parser

onigumo_feeder -- .json --> spider_operator

spider_operator ---> spider_materializer

onigumo_feeder -- .urls --> onigumo_downloader

onigumo_feeder -- .raw --> spider_parser

Suggested change

onigumo_feeder -- .json --> spider_operator

spider_operator ---> spider_materializer

onigumo_feeder -- .urls --> onigumo_downloader

onigumo_feeder -- .raw --> spider_parser

onigumo_feeder -- .json --> spider_operator

spider_operator ---> spider_materializer

onigumo_feeder -- .urls --> onigumo_downloader

onigumo_feeder -- .raw --> spider_parser

Glutexo · 2024-08-16T17:26:34Z

README.md

+    spider_operator ---> spider_materializer
+    onigumo_feeder  -- .urls --> onigumo_downloader
+    onigumo_feeder  -- .raw --> spider_parser
+
+    spider_parser     -. "<hash>.json" .->  spider_operator
+    onigumo_downloader -. "<hash>.raw"  .->  spider_parser
+    spider_operator     -. "<hash>.urls" .-> onigumo_downloader


Suggested change

spider_operator ---> spider_materializer

onigumo_feeder -- .urls --> onigumo_downloader

onigumo_feeder -- .raw --> spider_parser

spider_parser -. "<hash>.json" .-> spider_operator

onigumo_downloader -. "<hash>.raw" .-> spider_parser

spider_operator -. "<hash>.urls" .-> onigumo_downloader

onigumo_feeder -- .urls --> onigumo_downloader

onigumo_feeder -- .raw --> spider_parser

spider_parser -. "<hash>.json" .-> spider_operator

onigumo_downloader -. "<hash>.raw" .-> spider_parser

spider_operator -. "<hash>.urls" .-> onigumo_downloader

spider_operator ---> spider_materializer

Let’s push the spider_operator ---> spider_materializer relation to the bottom, so the Spider subgraph arrows are coupled together

agreed 🤝

Glutexo · 2024-08-16T17:46:13Z

README.md

+    start([START]) --> onigumo_feeder[FEEDER]
+    onigumo_feeder -- .raw --> Crawling
+    onigumo_feeder -- .urls --> Crawling
+    onigumo_feeder -- .json --> Crawling
+    Crawling --> spider_materializer(🕷️ MATERIALIZER)
+    spider_materializer --> done([END])


Suggested change

start([START]) --> onigumo_feeder[FEEDER]

onigumo_feeder -- .raw --> Crawling

onigumo_feeder -- .urls --> Crawling

onigumo_feeder -- .json --> Crawling

Crawling --> spider_materializer(🕷️ MATERIALIZER)

spider_materializer --> done([END])

start([START]) --> onigumo_feeder[FEEDER]

onigumo_feeder -- .raw --> Crawling

onigumo_feeder -- .urls --> Crawling

onigumo_feeder -- .json --> Crawling

Crawling --> spider_materializer(🕷️ MATERIALIZER)

spider_materializer --> done([END])

I‘d split this into smaller groups.

Glutexo · 2024-08-16T17:48:33Z

README.md

+    onigumo_feeder -- .json --> spider_operator
+    spider_operator ---> spider_materializer
+    onigumo_feeder -- .urls --> onigumo_downloader
+    onigumo_feeder -- .raw --> spider_parser
+
+    spider_parser -. "<hash>.json" .-> spider_operator
+    onigumo_downloader -. "<hash>.raw" .-> spider_parser
+    spider_operator -. "<hash>.urls" .-> onigumo_downloader


Suggested change

onigumo_feeder -- .json --> spider_operator

spider_operator ---> spider_materializer

onigumo_feeder -- .urls --> onigumo_downloader

onigumo_feeder -- .raw --> spider_parser

spider_parser -. "<hash>.json" .-> spider_operator

onigumo_downloader -. "<hash>.raw" .-> spider_parser

spider_operator -. "<hash>.urls" .-> onigumo_downloader

onigumo_feeder -- .json --> spider_operator

onigumo_feeder -- .urls --> onigumo_downloader

onigumo_feeder -- .raw --> spider_parser

spider_parser -. "<hash>.json" .-> spider_operator

onigumo_downloader -. "<hash>.raw" .-> spider_parser

spider_operator -. "<hash>.urls" .-> onigumo_downloader

spider_operator ---> spider_materializer

See my comment from my previous review. I’d push the spider_operator --→ spider_materializer relation to the bottom, so the Spider subgraph arrows are coupled together

…ations

Glutexo

It may be worth updating the description “The flowchart below illustrates the flow of data between those parts:” to use a plural form “flowchart” now when there are two. But it’s not necessary: we may consider that to be one flowchart consisting of two parts. 😃

Glutexo · 2024-08-16T17:55:53Z

README.md

+    onigumo_downloader -. "<hash>.raw" .-> spider_parser
+    spider_parser -. "<hash>.json" .-> spider_operator
+```
+


We were thinking about adding some title to the second diagram. But I don’t know what it should say.

Update mermaid graph of onigumo processing with a new approach

28b2680

nappex added 3 commits August 16, 2024 17:52

Switch fa icon of spider to emoji

ca1c368

Add another graph which differentiate spider and onigumo parts

26d560b

Remove title from graph

a710d41

Glutexo requested changes Aug 16, 2024

View reviewed changes

README.md Outdated Show resolved Hide resolved

Glutexo requested changes Aug 16, 2024

View reviewed changes

Fix spaces, same number of spaces everywhere

4ec3ee7

Glutexo requested changes Aug 16, 2024

View reviewed changes

nappex added 2 commits August 16, 2024 19:48

Move relation between operator and materializer to the end of the rel…

96969a7

…ations

Split the crawling graph to smaller group for better legibility

30065e1

Glutexo approved these changes Aug 16, 2024

View reviewed changes

Glutexo merged commit fc1b54e into Glutexo:master Aug 16, 2024
1 check passed

Glutexo mentioned this pull request Aug 16, 2024

Update description to match diagrams #239

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update mermaid graph of onigumo processing with a new approach #237

Update mermaid graph of onigumo processing with a new approach #237

nappex commented Aug 3, 2024

nappex commented Aug 3, 2024 •

edited

Loading

Glutexo left a comment

Glutexo left a comment

Glutexo Aug 16, 2024

Glutexo Aug 16, 2024

Glutexo Aug 16, 2024

Glutexo Aug 16, 2024

Glutexo Aug 16, 2024

nappex Aug 16, 2024

Glutexo Aug 16, 2024

Glutexo Aug 16, 2024

Glutexo left a comment

Glutexo Aug 16, 2024

		spider_parser(🕷️ PARSER)
		spider_operator(🕷️ OPERATOR)

Update mermaid graph of onigumo processing with a new approach #237

Update mermaid graph of onigumo processing with a new approach #237

Conversation

nappex commented Aug 3, 2024

nappex commented Aug 3, 2024 • edited Loading

Glutexo left a comment

Choose a reason for hiding this comment

Glutexo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Glutexo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nappex commented Aug 3, 2024 •

edited

Loading