Merge pull request #32 from kmlbgn/master

feat: major refactoring + link between tabs support

README.md

@@ -1,14 +1,22 @@
-# Description
-KIRA's documentation system integrates with docu-notion-kira, a forked version of [docu-notion](https://github.com/sillsdev/docu-notion) tailored for Kira Network. 
+# Nocusorus
+Nocusorus is a tool that fetches content from a provided Notion root page and generates a structured folder of markdown-based files containing its content for Docusaurus. Nocusaurus is built from [docu-notion](https://github.com/sillsdev/docu-notion).
 
-Docu-notion allows the use of Notion as the primary editing platform to produce content suitable for static site generators; Docusaurus in this case. This combination meets several challenging requirements, such as automatique deployment, editing workflow features, localization support via Crowdin, and capabilities for both online and offline distribution. Future plans include adding versioning capabilities.
+The root page in Notion serves as the foundation for the generated documentation structure. It contains one or more "Tabs Pages", each representing a separate folder structure or content hierarchy, similar to a merkle tree or a separate navigation section.
 
-# How It Works ?
+Cross-linking between different Tab structures is supported, allowing for interconnected documentation across multiple content hierarchies.
 
-Docu-notion fetches content from a provided Notion root page and produce a structured folder of markdown-base files of its content. The root page has two main components:
+Nocusaurus supports custom parsing through plugins, making it versatile for use with various static site generators beyond Docusaurus. This flexibility allows you to create documentation websites or other projects using the static generator of your choice while still leveraging Notion as CMS.
 
-1. **The Database (Optional)** - This is where the documentation pages are stored. They include content and are equipped with workflow properties to facilitate a Kanban-style management process where pages can have metadata that can be leveraged and are published according to their ‘status’.
-2. **The Outline Page (Mandatory)** - This is a central Notion page that organizes content hierarchically. It serves as the foundation of the documentation structure. The arrangement of sub-pages within the Outline is directly reflected in the final documentation site and its sidebar navigation. These sub-pages should link back to the relevant documents housed in the database.
+# How It Works
+Nocusorus requires a Notion root page with two main components:
+
+1. Tabs Page (Mandatory)
+A Tab page is a Notion page that organizes content hierarchically. Each Tab page represents a separate folder structure or content hierarchy in the generated documentation. The arrangement of sub-pages within a Tab is directly reflected in the final documentation and the Tab's sidebar navigation. The Tab's sub-pages link back to the relevant documents housed in the database (if applicable).
+
+For example, in a Docusaurus website using multi-instances, each Tab page could represent the foundation for the "docs", "blog", or other other top navigation sections.
+
+2. Databases (Optional)
+Databases in Notion can be used to store the documentation pages within a Tab. When pages are stored in a database, they can have metadata that can be leveraged, and they are published according to their 'status'. This enables a Kanban-style workflow management process for the documentation pages.
 
 ### **Page Structure in the Outline**
 
@@ -23,11 +31,11 @@ Blocks listed under the Outline page can be of the following types:
 
 ### **Links**
 
-Docu-notion automatically identifies and removes blocks that are either child pages or links to pages located at the root level of the page. If you need to include such blocks within your content, they must be embedded within another block type, like a table or a column, or they should be accompanied by some text within the same block to trick this logic.
+Nocusorus automatically identifies and removes blocks that are either child pages or links to pages located at the root level of the page. If you need to include such blocks within your content, they must be embedded within another block type, like a table or a column, or they should be accompanied by some text within the same block to trick this logic.
 
 # **Custom Pages**
 
-Docusaurus automatically generates custom pages from the `src/pages` directory, creating corresponding slugs and links. You can create any page within the root page in Notion, for instance naming it "src/page", and manually move these pages into the Docusaurus `src/pages` folder as needed. This approach is simpler and easily managed using github workflow.
+Docusaurus automatically generates custom pages from the `src/pages` directory, creating corresponding slugs and links. You can create any page within the root page in Notion, for instance naming it "src/page", and manually move these pages into the Docusaurus `src/pages` folder as needed. This approach is simpler and easily managed using github workflow or your own terminal.
 
 **Note on Conflicts**: Pages within `src/pages` are prioritized by Docusaurus and can lead to conflicts with pages that have matching slugs elsewhere in the project. E.g. If both an index.md or a page with "/" slug in the main documentation and an "index.js" in `src/pages` exist, Docusaurus will prioritize the content in `src/pages`, potentially overlooking the index.md.
 
@@ -48,7 +56,7 @@ To map Notion callouts to Docusaurus admonitions, ensure the icon is for the typ
 
 The default admonition type, if no matching icon is found, is "note".
 
-# Setup: Docu-notion-kira + docusaurus
+# Setup: Nocusorus + docusaurus
 
 #### Host specs:
 
@@ -124,7 +132,7 @@ Ubuntu 20.04
     export DOCU_NOTION_SAMPLE_ROOT_PAGE=[***]
     export DOCU_NOTION_INTEGRATION_TOKEN=[***]
     ```
-  * Go to the root page and add docu-notion-kira integration. This page should have, as direct children, "Outline" (required) and "Database" (optional) pages. Follow these instructions. Source: [Notion integration](https://developers.notion.com/docs/create-a-notion-integration#give-your-integration-page-permissions)
+  * Go to the root page and add nocusaurus integration. This page should have, as direct children, a Tab (required) and a "Database" associated to this tab (optional). Follow these instructions. Source: [Notion integration](https://developers.notion.com/docs/create-a-notion-integration#give-your-integration-page-permissions)
 
 3. **Install Dependencies:**
   ```bash
@@ -134,7 +142,7 @@ Ubuntu 20.04
 4. **Parse Pages with docu-notion:**
 
   ```bash
-  npx docu-notion-kira -n $DOCU_NOTION_INTEGRATION_TOKEN -r $DOCU_NOTION_SAMPLE_ROOT_PAGE
+  npx nocusaurus -n $DOCU_NOTION_INTEGRATION_TOKEN -r $DOCU_NOTION_SAMPLE_ROOT_PAGE
   ```
 
 ## Starting Docusaurus Server
@@ -146,9 +154,9 @@ Ubuntu 20.04
   ```
   * Source [Docusaurus Intallation Guide](https://docusaurus.io/docs/installation)
 
-# Docu-notion Command line
+# Nocusorus Command line
 
-Usage: docu-notion-kira -n <token> -r <root> [options]
+Usage: nocusaurus -n <token> -r <root> [options]
 
 Options:
 

src/HierarchicalNamedLayoutStrategy.ts

@@ -1,21 +1,11 @@
 import * as fs from "fs-extra";
 import sanitize from "sanitize-filename";
 import { LayoutStrategy } from "./LayoutStrategy";
-import { NotionPage, PageSubType } from "./NotionPage";
-import {
-  endGroup,
-  error,
-  group,
-  info,
-  logDebug,
-  verbose,
-  warning,
-} from "./log";
+import { NotionPage } from "./NotionPage";
 
 // This strategy gives us a file tree that mirrors that of notion.
-// Each level in the outline becomes a directory, and each file bears the name of the Notion document.
+// Each level in the outline becomes a directory, and each file bears the name of the Notion page.
 // As long as you use slugs, the urls is still just something like https://site/slug
-
 export class HierarchicalNamedLayoutStrategy extends LayoutStrategy {
   public newLevel(
     dirRoot: string,

src/LayoutStrategy.ts

@@ -1,25 +1,13 @@
 import * as fs from "fs-extra";
-import { verbose } from "./log";
 import { NotionPage } from "./NotionPage";
 
 // Here a fuller name would be File Tree Layout Strategy. That is,
 // as we walk the Notion outline and create files, where do we create them, what do we name them, etc.
 export abstract class LayoutStrategy {
   protected rootDirectory = "";
-  protected existingPagesNotSeenYetInPull: string[] = [];
 
   public setRootDirectoryForMarkdown(markdownOutputPath: string): void {
     this.rootDirectory = markdownOutputPath;
-    this.existingPagesNotSeenYetInPull =
-      this.getListOfExistingFiles(markdownOutputPath);
-  }
-
-  public async cleanupOldFiles(): Promise<void> {
-    // Remove any pre-existing files that aren't around anymore; this indicates that they were removed or renamed in Notion.
-    for (const p of this.existingPagesNotSeenYetInPull) {
-      verbose(`Removing old doc: ${p}`);
-      await fs.rm(p);
-    }
   }
 
   public abstract newLevel(
@@ -39,12 +27,6 @@ export abstract class LayoutStrategy {
     return ("/" + page.slug).replaceAll("//", "/");
   }
 
-  public pageWasSeen(page: NotionPage): void {
-    const path = this.getPathForPage(page, ".mdx");
-    this.existingPagesNotSeenYetInPull =
-      this.existingPagesNotSeenYetInPull.filter(p => p !== path);
-  }
-
   protected getListOfExistingFiles(dir: string): string[] {
     return fs.readdirSync(dir).flatMap(item => {
       const path = `${dir}/${item}`;

src/plugins/internalLinks.ts

@@ -1,5 +1,5 @@
 import { IDocuNotionContext, IPlugin } from "./pluginTypes";
-import { error, warning, verbose } from "../log";
+import { warning, verbose } from "../log";
 import { NotionPage } from "../NotionPage";
 
 // converts a url to a local link, if it is a link to a page in the Notion site
@@ -19,9 +19,9 @@ export function convertInternalUrl(
     return undefined;
   }
   const id = match[1];
-  const pages = context.pages;
+  const allTabsPages = context.allTabsPages;
   // find the page where pageId matches hrefFromNotion
-  const targetPage = pages.find(p => {
+  const targetPage = allTabsPages.find(p => {
     return p.matchesLinkId(id);
   });
 
@@ -70,9 +70,9 @@ function convertInternalLink(
       hrefFromNotion = hrefFromNotion.substring(lastSpecialCharIndex + 1);
   }
 
-  const pages = context.pages;
+  const allTabsPages = context.allTabsPages;
   // find the page where pageId matches hrefFromNotion
-  const targetPage = pages.find(p => {
+  const targetPage = allTabsPages.find(p => {
     return p.matchesLinkId(hrefFromNotion);
   });
 

src/plugins/pluginTestRun.ts

@@ -13,7 +13,8 @@ import { numberChildrenIfNumberedList } from "../pull";
 export async function blocksToMarkdown(
   config: IDocuNotionConfig,
   blocks: NotionBlock[],
-  pages?: NotionPage[],
+  tabsPages?: NotionPage[],
+  allTabsPages?: NotionPage[],
   // Notes on children:
   //   - These children will apply to each block in blocks. (could enhance but not needed yet)
   //   - If you are passing in children, it is probably because your parent block has has_children=true.
@@ -28,10 +29,6 @@ export async function blocksToMarkdown(
     notionClient,
   });
 
-  // if (pages && pages.length) {
-  //   console.log(pages[0]);
-  //   console.log(pages[0].matchesLinkId);
-  // }
   const docunotionContext: IDocuNotionContext = {
     config: config,
     notionToMarkdown: notionToMD,
@@ -61,7 +58,8 @@ export async function blocksToMarkdown(
       imgPrefixInMarkdown: "",
       statusTag: "",
     },
-    pages: pages ?? [],
+    tabsPages: tabsPages ?? [],
+    allTabsPages: allTabsPages ?? [],
     counts: {
       output_normally: 0,
       skipped_because_empty: 0,
@@ -89,9 +87,13 @@ export async function blocksToMarkdown(
     // },
   };
 
-  if (pages && pages.length) {
-    console.log(pages[0].matchesLinkId);
-    console.log(docunotionContext.pages[0].matchesLinkId);
+  if (tabsPages && tabsPages.length) {
+    console.log(tabsPages[0].matchesLinkId);
+    console.log(docunotionContext.tabsPages[0].matchesLinkId);
+  }
+  if (allTabsPages && allTabsPages.length) {
+    console.log(allTabsPages[0].matchesLinkId);
+    console.log(docunotionContext.tabsPages[0].matchesLinkId);
   }
   const r = await getMarkdownFromNotionBlocks(
     docunotionContext,
@@ -278,6 +280,7 @@ export async function oneBlockToMarkdown(
   return await blocksToMarkdown(
     config,
     [fullBlock as NotionBlock],
-    targetPage ? [dummyPage1, targetPage, dummyPage2] : undefined
+    targetPage ? [dummyPage1, targetPage] : undefined,
+    targetPage ? [dummyPage1, targetPage, dummyPage2] : undefined,
   );
 }

src/plugins/pluginTypes.ts

@@ -74,8 +74,9 @@ export type IDocuNotionContext = {
   notionToMarkdown: NotionToMarkdown;
   directoryContainingMarkdown: string;
   relativeFilePathToFolderContainingPage: string;
+  tabsPages: NotionPage[];
+  allTabsPages: NotionPage[];
   convertNotionLinkToLocalDocusaurusLink: (url: string) => string | undefined;
-  pages: NotionPage[];
   counts: ICounts;
 
   // If the output is creating things like react elements, you can append their import definitions