Feature suggestion: use custom filter to split sections

[Gk0Wk]

Hello! Earlier in the TiddlyWiki Talk, I mentioned that I would like to be able to segment tiddlers not only based on first and second-level headers, but also in a more flexible way. After that, I tried to write a custom filter to achieve this.

First, add a new tiddler to define the new filter:

/*\
title: $:/core/modules/filters/sectionsplit.js
type: application/javascript
module-type: filteroperator
author: Gk0WK(nmg_wk @yeah.net)

Split sections of wikitext

\*/
(function() {

    /*jslint node: true, browser: true */
    /*global $tw: false */
    "use strict";

    /*
    Export our filter function
    */
    exports.sectionsplit = function(source, operator, options) {
        var options = {};
        operator.operand.split('+').forEach(option => {
            options[option] = true;
        })
        var results = [];
        source(function(tiddler, title) {
            var section = [];
            title.split('\n').forEach(line => {
                var newSection = false;

                if ((options.all || options.h1) && /^!\s+[^\n]+$/.test(line)) {
                    newSection = true;
                } else if ((options.all || options.h2) && /^!!\s+[^\n]+$/.test(line)) {
                    newSection = true;
                } else if ((options.all || options.h3) && /^!!!\s+[^\n]+$/.test(line)) {
                    newSection = true;
                } else if ((options.all || options.h4) && /^!!!!\s+[^\n]+$/.test(line)) {
                    newSection = true;
                } else if ((options.all || options.h5) && /^!!!!!\s+[^\n]+$/.test(line)) {
                    newSection = true;
                } else if ((options.all || options.h6) && /^!!!!!!\s+[^\n]+$/.test(line)) {
                    newSection = true;
                } else if ((options.all || options.hr) && /^---+$/.test(line)) {
                    newSection = true;
                } else if ((options.all || options.blank) && /^\s*$/.test(line)) {
                    newSection = true;
                }

                if (newSection && section.length > 0) {
                    results.push(section.join('\n'));
                    section = [];
                }
                section.push(line);
            });
            results.push(section.join('\n'));
        });
        return results;
    };

})();

This toddler also has:

• type: application/javascript
• module-type field: filteroperator
• title: $:/core/modules/filters/sectionsplit.js

Then, change the filter expression in line 11 of source/section/macros/main.tid to [<sourceText>sectionsplit[h1+h2+h3]!is[blank]regexp<nonWhitespace>].

Then TW-Section can split toddler by h1, h2 and h3. You can change it to [h1+h2+h3+h4] and so on to take more split rules. Now this filter support:

• h1 ~ h6
• hr
• blank, means blank line
• all, means all above

If you think this filter is useful, I can add more rules to it. 😄

[pmario]

if ((options.all || options.h1) && /^!\s+[^\n]+$/.test(line)) {

That's not how the TW parser tests for headings. See: !!.myStyle heading text comes here and the whitespace after !! is optional.

[kookma]

Hello! Earlier in the TiddlyWiki Talk, I mentioned that I would like to be able to segment tiddlers not only based on first and second-level headers, but also in a more flexible way. After that, I tried to write a custom filter to achieve this.
Then, change the filter expression in line 11 of source/section/macros/main.tid to [<sourceText>sectionsplit[h1+h2+h3]!is[blank]regexp<nonWhitespace>].

Thank you, I welcome your contribution! Unfortunately I know little JS. but this is a nice filter if it can accept different rules!
Would you please also add rules to split on <tag ...> contents </tag> a tag can be section, or article or other sematic html5 tag!

Then TW-Section can split toddler by h1, h2 and h3. You can change it to [h1+h2+h3+h4] and so on to take more split rules. Now this filter support:

• h1 ~ h6
• hr
• blank, means blank line
• all, means all above

does this means I can pass sectionsplit[h1+h2] and split only on heading one and two?

Also as @pmario said, in Tiddlywiki we have !!.myStyle heading text comes here which means you can apply styles through css classes! Would you please correct the filter to handle such cases!

If you think this filter is useful, I can add more rules to it. 😄

Yes, this is absolutely useful!

Thank you

[Gk0Wk]

That's not how the TW parser tests for headings. See: !!.myStyle heading text comes here and the whitespace after !! is optional.

@pmario Thank you! I have changed the expressions of header checking.

does this means I can pass sectionsplit[h1+h2] and split only on heading one and two?

@kookma Yes, sectionsplit[] (but useless), sectionsplit[h2], sectionsplit[h1+h2], sectionsplit[hr+h3+h1] can also work. The order and number of rules do not matter.

---

It is possible to split tiddler with HTML tags, but there are some problems:

• I need to write a state machine to find opened/closed tags, the regex of </[^>]+>/gim can work on most HTML text but some special ones like <tag attr=">"></tag>.
• The HTML section may not start at the head of the line, and may not end at the tail of the line. For example:

* Hello <div>
World
</div>!

• The HTML sections may not close or close wrongly, how to solve them?

So, I think there are some questions for us to discuss before adding this important rule.

---

I also have another question. How does the plugin work? If I remove some blank lines and trim returned text (for example, \n123\n -> 123), can the plugin work in the proper way?

Thanks all.

---

Updated js code:

JS Code

/*\
title: $:/core/modules/filters/sectionsplit.js
type: application/javascript
module-type: filteroperator
creator: Gk0WK(nmg_wk @yeah.net)

Split sections of wikitext

\*/
(function() {

    /*jslint node: true, browser: true */
    /*global $tw: false */
    "use strict";

    /*
    Export our filter function
    */
    exports.sectionsplit = function(source, operator, options) {
        var splitOptions = {};
        operator.operand.split('+').forEach(option => {
            splitOptions[option] = true;
        })
        var results = [];
        source(function(tiddler, title) {
            var section = [];
            title.split('\n').forEach(line => {
                var newSection = false;

                if ((splitOptions.all || splitOptions.h1) && /^!(\.[\w\-]+)?\s+[^\n]+$/.test(line)) {
                    newSection = true;
                } else if ((splitOptions.all || splitOptions.h2) && /^!!(\.[\w\-]+)?\s+[^\n]+$/.test(line)) {
                    newSection = true;
                } else if ((splitOptions.all || splitOptions.h3) && /^!!!(\.[\w\-]+)?\s+[^\n]+$/.test(line)) {
                    newSection = true;
                } else if ((splitOptions.all || splitOptions.h4) && /^!!!!(\.[\w\-]+)?\s+[^\n]+$/.test(line)) {
                    newSection = true;
                } else if ((splitOptions.all || splitOptions.h5) && /^!!!!!(\.[\w\-]+)?\s+[^\n]+$/.test(line)) {
                    newSection = true;
                } else if ((splitOptions.all || splitOptions.h6) && /^!!!!!!(\.[\w\-]+)?\s+[^\n]+$/.test(line)) {
                    newSection = true;
                } else if ((splitOptions.all || splitOptions.hr) && /^---+$/.test(line)) {
                    newSection = true;
                } else if ((splitOptions.all || splitOptions.blank) && /^\s*$/.test(line)) {
                    newSection = true;
                }

                if (newSection && section.length > 0) {
                    results.push(section.join('\n').trim());
                    section = [];
                }
                section.push(line);
            });
            results.push(section.join('\n').trim());
        });
        return results;
    };

})();

[kookma]

@Gk0Wk thank you!
I decided to keep the simple version only a wikitext plugin (without JS). Then we can have subplung or advanced SE to include the JS and more sophisticated parsing!

I think for now

• only sectionize the part started by a heading (e.g. !, !!, etc.)
• ignore the part at the beginning of tidler with no heading

Next

• support for <section>...</section>
• OR some custom tag like <chapter>....</chapter>

but not any tag!

[Gk0Wk]

Hi @kookma !

I suddenly had a better idea to use TiddlyWiki's internal syntax parser to generate a document parse instead of writing one myself. This approach should allow for easier and more flexible segmentation (including the features mentioned above). I'll try this approach when I have time.

[kookma]

That would be great! I appreciate if you could give me a short working example!

[Gk0Wk]

@kookma Hi. I'm sorry to inform you that the TiddlyWiki core parser does not parse results with substring location information, which makes the syntax tree unusable for splitting sections. I think a better way is still to write a separate syntax parser. As it happens, I am implementing such a parser in my TW5-CodeMirror-Enhanced project, and if I finish a simple parser I will try to use it in TW_Section.

BTW, I encountered some problems in using TW_Section, which are supposed to be caused by overly simple regular expressions. TW_Section does not correctly divide and render the following two cases:

case 1:

<<<
! a
<<<

case 2:

```
! a
```

[kookma]

I am implementing such a parser in my TW5-CodeMirror-Enhanced project, and if I finish a simple parser I will try to use it in TW_Section.

Lovely, I am sure the TW5-CodeMirror-Enhanced will receive a lot of attention, as it modernize the editor and simplify the writing in TW.

BTW, I encountered some problems in using TW_Section, which are supposed to be caused by overly simple regular expressions. TW_Section does not correctly divide and render the following two cases:

Yes, I am aware of this and will add this in documentation!
Another issue is with local macros! If you sectionize a tiddler with local macro, then macro do not works in other sections!

That is why I raised the definition of tiddlers in Talk Tiddlywiki! Section Editor best works for plain tiddler!

<<<
! a
<<<

do you have a workaround for this? In documentation I recommend to use <h1>... but this is a temporary solution.

[kookma]

@Gk0Wk - now we have TiddlyWiki/TiddlyWiki5#4977 merged!
do you think we can now extract by tags!
For example extract all sections (<section class= style=> ....</section>)

[Gk0Wk]

@kookma Sure, we can. I've done some experimentation and even without this updated parser, the text can be properly sectioned.

The question is, how do I get this information to you? I think you need information that includes at least substrings, start and end position information, and the type of section (e.g. title or paragraph), but the filter only returns a number of strings, which is confusing to me.

[Gk0Wk]

I'm not good at writing filters. Can filter return more information as meta data?

[Gk0Wk]

Another question is, if sections are not necessarily divided by titles, then for sections that are not divided by titles, what are their titles?

[kookma]

@Gk0Wk - sorry for late reply!
What I thought for Sectionizing here is to convert the text into editable blocks! So, we do not add any extra text/heading/title.
The code just distinguish each section for example <section>...</section> then a small icon/button (as superscript or with some offset above the text) at the beginning of each section is appeared (it can be displayed always or just on mouse hover). On clicking the icon/button, the section is rendered in edit mode in place! on done the section is appeared in view mode!

[Gk0Wk]

@kookma It doesn't matter, I'm busy too. lol

I modified the code of the filter according to your suggestion and now it works properly as follows.

[[SomeTiddler]get[text]sectionsplit[]]

This filter takes the text of the tiddler and splits the text according to certain rules. if the rule is empty it means that all rules are used. The currently supported rules are as follows.

• all: all rules.
• h1, h2, h3, h4, h5, h6, h*: headers, where h* represents all levels of headers.
• p: text paragraphs.
• ul: Bullet lists.
• ol: Number lists.
• dl: Defines and descriptions.
• list: ul + ol + dl.
• tiddler: Transcludes.
• table: Tables.
• section: Segments wrapped by <section></section> tags.
• latex: LaTeX blocks.
• hr: Dividing Line.

Usage: Separate the rule names with + and place them inside [].

sectionsplit[list+h*+section]

And the filter will return all substrings that are split.

Note: The first substring returned by the filter is always the parameter part at the beginning of the tiddler (consisting of \define, \rules, \import, \whitespace, etc.); if there is no parameter part, then it is an empty string. This is because I believe that the parameters section is something that may be used in each section, so it should be taken out separately and placed at the beginning of each section when displayed (but should not appear when edited, but can be edited separately).

---

I have applied for a PR, which contains the latest filter scripts.

[AnthonyMuscio]

In the section names listed, only all, h* and latex (?) are in fact not html tags, perhaps these exceptions could be coded an allow any arbitrary html tag be used. The code size will reduce and the functionality increase.

Question, can it handle nested sections?

eg;

<section>
top
<section>
middle
</section>
botton
</section>

and what happens?

[kookma]

Thank you @Marxsal. There are some issues and there is missing code to include this in the UI
However I am thinking to use sectionsplit as more general purpose sectionizer! see above discussion!

I highly welcome your contribution and help!

[kookma]

Question, can it handle nested sections?

No! but it is a good improvement.

[AnthonyMuscio]

Is this proceeding, it would be a great feature in the next release?

[kookma]

No, May be @Gk0Wk wants to implement this.

[AnthonyMuscio]

Of course a custom filter operator can be delivered as a package or plugin?

• Perhaps this is specific enough to be available.

[Marxsal]

I'm thinking that the approach used in the first Post, once expanded, might be more compatible between versions of TW?

[Gk0Wk]

A new feature is soon to be merged, which will map the start and end positions in the text for each section of the complete Wiki syntax tree. Therefore, this should be feasible in the future.

start, end and substring of a tiddler:

[kookma]

Hi @Gk0Wk
With new features of TW 5.3.3, your JS solution above for sectionizing is more demanding!
Would you please update your solution and share it here! I am thinking to switch from current wikitext solution to your JS one in TW-Section 2.

[Gk0Wk]

It seems that my feature has not been merged yet (in v5.3.3); I am continuing to wait for the release of a new version. I have already rewritten part of the core and implemented a demo of a block editor, which you can test here: https://gk0wk.github.io/TiddlySeq/#TestForSectionEditor

[linonetwo]

Maybe I can use this to open block level WYSIWYG editor on hover?

WYSIWYG editor's rendering is the same as the render mode, so user won't even notice the change between edit mode and render mode. Only when encounter complex wikiscript it show a edit button like section editor.