When using the current HTML tokenizer, it submits two tokens here for the HTML renderer: <cite>, </cite>
\nI want to be able to recognize <cite>marked</cite> as a single token.
My current approach is this
\nexport class CiteTokenizer extends Tokenizer {\n constructor(options?: MarkedOptions) {\n super(options);\n }\n\n override html(src: string): Tokens.HTML | undefined {\n const match = /^<cite>([\\s\\S]+?)<\\/cite>/.exec(src);\n\n if (match) {\n return {\n type: 'html',\n pre: false,\n text: match[0],\n block: false,\n raw: match[0],\n };\n }\n\n return super.html(src);\n }\n}This does work for single lines of cite elements, but not if other text is in front of it. No matter what I tried so far, I did not succeed in implementing such behavior.
Is this even possible by implementing a tokenizer, or is a different way a better approach?
","upvoteCount":1,"answerCount":1,"acceptedAnswer":{"@type":"Answer","text":"You will need to write a custom extension.
\nSomething like:
\nconst citeExtension = {\n name: 'cite',\n level: 'inline',\n start(src) { return src.indexOf(\"<cite\"); },\n tokenizer(src, tokens) {\n const rule = /^<cite>([\\s\\S]+?)<\\/cite>/;\n const match = rule.exec(src);\n if (match) {\n return {\n type: 'html',\n pre: false,\n text: match[0],\n block: false,\n raw: match[0],\n };\n }\n }\n};
-
|
Hello, I am trying to build a custom tokenizer because I want to achieve the following behavior: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Fusce erat quam, ultrices at diam a, ullamcorper commodo sapien. Curabitur vestibulum auctor massa sed placerat. Orci varius natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Vivamus sed consequat orci. <cite>marked</cite>When using the current HTML tokenizer, it submits two tokens here for the HTML renderer: My current approach is this export class CiteTokenizer extends Tokenizer {
constructor(options?: MarkedOptions) {
super(options);
}
override html(src: string): Tokens.HTML | undefined {
const match = /^<cite>([\s\S]+?)<\/cite>/.exec(src);
if (match) {
return {
type: 'html',
pre: false,
text: match[0],
block: false,
raw: match[0],
};
}
return super.html(src);
}
}This does work for single lines of Is this even possible by implementing a tokenizer, or is a different way a better approach? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 2 replies
-
|
You will need to write a custom extension. Something like: const citeExtension = {
name: 'cite',
level: 'inline',
start(src) { return src.indexOf("<cite"); },
tokenizer(src, tokens) {
const rule = /^<cite>([\s\S]+?)<\/cite>/;
const match = rule.exec(src);
if (match) {
return {
type: 'html',
pre: false,
text: match[0],
block: false,
raw: match[0],
};
}
}
}; |
Beta Was this translation helpful? Give feedback.
-
|
Thank you very much for this answer!! I sadly cannot seem to get this extension triggered. Neither the start nor tokenizer function gets called in my code, while my previous tokenizer does get called. For now I am generating my options like this using the extension: export const markedOptionsFactory = (): MarkedOptions => {
return {
extensions: [citeExtension] as any,
};
};
const citeExtension = {
name: 'cite',
level: 'inline',
start(src: any) {
console;
return src.indexOf('<cite');
},
tokenizer(src: any, tokens: any) {
const rule = /^<cite>([\s\S]+?)<\/cite>/;
const match = rule.exec(src);
console.log(src, tokens);
if (match) {
return {
type: 'html',
pre: false,
text: match[0],
block: false,
raw: match[0],
};
}
return;
},
renderer(token: any) {
return `<cite>${token.text}</cite>`;
},
};Am I missing something for it to be recognized? Thank you very much again!! |
Beta Was this translation helpful? Give feedback.
-
|
This seems to work import { Marked } from 'marked';
const marked = new Marked();
const markedOptionsFactory = () => {
return {
extensions: [citeExtension]
};
};
const citeExtension = {
name: 'cite',
level: 'inline',
start(src) {
return src.indexOf('<cite');
},
tokenizer(src, tokens) {
const rule = /^<cite>([\s\S]+?)<\/cite>/;
const match = rule.exec(src);
if (match) {
return {
type: 'cite',
raw: match[0],
text: match[0]
};
}
},
renderer(token) {
return `<cite id="test">${token.text}</cite>`;
}
};
marked.use(markedOptionsFactory());
console.log(marked.parse('<cite>foo</cite>')); |
Beta Was this translation helpful? Give feedback.
You will need to write a custom extension.
Something like: