Skip to content

Add wikify operator#8730

Open
Jermolene wants to merge 5 commits intomasterfrom
wikify-operator
Open

Add wikify operator#8730
Jermolene wants to merge 5 commits intomasterfrom
wikify-operator

Conversation

@Jermolene
Copy link
Member

This PR adds a "wikify" operator that has the same functionality as the wikify widget. I regard this as quite controversial because while it unlocks some important capabilities, it also suffers from performance issues and invites users to adopt patterns that can be dangerous:

  • The performance issues arise from the fact that the wikify operator does not allow us to use the usual mechanism for caching parsed tiddlers. This means that to evaluate the operator always requires a full parse of the input text as well as the construction of a widget tree (which would always be required, even if caching were to be implemented somehow)
  • The issue with dangerous patterns is that users might be tempted to replace complex filters with a wikitext equivalent, for example using the <$list> widget to generate a set of numbers instead of the [range[]] operator. Besides being horribly inefficient, there are pitfalls with unexpected wikification

This PR is cherry picked from #8702; I plan to refactor the work there to not require the wikify operator because of the performance issues.

The question now is what are the use cases for the wikify operator, and are there any other ways to accomplish those use cases?

@github-actions
Copy link

github-actions bot commented Nov 7, 2024

Confirmed: Jermolene has already signed the Contributor License Agreement (see contributing.md)

@netlify
Copy link

netlify bot commented Nov 7, 2024

Deploy Preview for tiddlywiki-previews ready!

Name Link
🔨 Latest commit a812313
🔍 Latest deploy log https://app.netlify.com/projects/tiddlywiki-previews/deploys/699d84d4c84e910008791e0c
😎 Deploy Preview https://deploy-preview-8730--tiddlywiki-previews.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@Jermolene Jermolene mentioned this pull request Nov 7, 2024
14 tasks
@Leilei332
Copy link
Contributor

Leilei332 commented Nov 7, 2024

When updating the flexoki palette, I found that the tag won't display properly unless index background, foreground and tag-background 's value is hardcoded. So the wikify operator should be used in the tag macro.

But we had better not use it at present since tag pill is often used in UI, which may cause performance issues.

@Jermolene
Copy link
Member Author

When updating the flexoki palette, I found that the tag won't display properly unless index background, foreground and tag-background 's value is hardcoded. So the wikify operator should be used in the tag macro.

Ouch. I suspect that the compiled palette idea under investigation in #8702 will resolve a number of issues like this, and ultimately should allow us to significantly improve the stylesheet refresh performance.

@github-actions
Copy link

github-actions bot commented Jun 3, 2025

📊 Build Size Comparison: empty.html

Branch Size
Base (master) 2528.4 KB
PR 2530.6 KB

Diff: ⬆️ Increase: +2.2 KB

@CrossEye
Copy link
Contributor

CrossEye commented Jun 3, 2025

I agree that this could be quite useful... at times.

I also agree that there are all sorts of perils in including it.

I wonder if TW could reasonably adopt something akin to JS's strict mode where some unobtrusive signal allows the user to use certain features. That is, could we have wikify unavailable but available if the text of, say, $:/temp/advanced-features/ included [[Wikify Operator]]. That might require a StartupAction to trigger this, which might be a decent barrier against unnecessary usage. Or we could put this in a non-tenp namespace. And of course, since we have our own parser, we could use the same String-at-the-top mechanism of use strict, which allows greater control over scoping of the feature.

This may just be crazy talk, but it seems like an interesting possibility.

@uzvg
Copy link

uzvg commented Feb 24, 2026

The question now is what are the use cases for the wikify operator, and are there any other ways to accomplish those use cases?

I’m currently working on a flashcards plugin for TiddlyWiki that integrates with Anki. In this workflow, wikitext content needs to be wikified (to HTML) and then embedded into a JSON payload sent via HTTP.

In this scenario, the wikify operator is very useful. It provides two concrete advantages:

  1. Avoiding repeated wikification

Without the operator, I would need to wikify content multiple times at different stages:

  • First, wikify the flashcard content (wikitext → HTML).
  • Then embed that HTML into a structured JSON payload, which often requires another wikify step.

This introduces redundant parsing/rendering overhead.

With the wikify operator, wikification can be done once at the first stage. The rest of the transformation (such as building the JSON structure) can be handled using functions or procedures and then imported as needed.

  1. Improved composability inside filter pipelines

Being able to wikify directly within a filter allows the transformation to remain declarative and self-contained. This reduces the need for additional widgets or procedural workarounds, which helps keep both performance and code complexity under control.

It is especially useful when composing JSON payloads, where having wikification available inside the filter pipeline enables a cleaner data transformation flow and avoids unnecessary rendering passes.

If performance remains a concern, perhaps the documentation could clarify that this operator should be used thoughtfully in performance-sensitive loops, rather than discouraging the feature entirely.

@github-actions
Copy link

github-actions bot commented Feb 24, 2026

📊 Build Size Comparison: empty.html

Branch Size
Base (master) 2486.7 KB
PR 2488.8 KB

Diff: ⬆️ Increase: +2.2 KB


⚠️ Change Note Status

This PR appears to contain code changes but doesn't include a change note.

Please add a change note by creating a .tid file in editions/tw5.com/tiddlers/releasenotes/<version>/

📚 Documentation: Release Notes and Changes

💡 Note: If this is a documentation-only change, you can ignore this message.

@Jermolene
Copy link
Member Author

Thanks @uzvg that is a cogent and compelling argument. I've taken this out of draft, resolved the merge conflict, and marked it for consideration for v5.5.0.

@Jermolene Jermolene marked this pull request as ready for review February 24, 2026 10:46
@Jermolene
Copy link
Member Author

@pmario there are a lot of eslint errors for this PR. I can see that they are concerned with indentation, but there is no indication that I can find of which file is affected. Am I perhaps missing something?

Comment on lines +9 to +12
(function(){

/*jslint node: true, browser: true */
/*global $tw: false */
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Function wrappers should be removed now. I believe they're causing indentation errors.

@saqimtiaz
Copy link
Member

@Jermolene removing the function wrappers in the two new JavaScript modules should resolve the lint errors for indentation.

@Jermolene
Copy link
Member Author

Doh! Thank you @Leilei332 @saqimtiaz.

@uzvg
Copy link

uzvg commented Feb 24, 2026

Thanks, @Jermolene — I really appreciate it.

@saqimtiaz saqimtiaz moved this to Needs feedback in Planning for v5.5.0 Feb 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Needs feedback

Development

Successfully merging this pull request may close these issues.

5 participants