Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor command-line handling #335

Draft
wants to merge 7 commits into
base: main
Choose a base branch
from
Draft

Refactor command-line handling #335

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
5395a66
Reworking CLI tokenization
Yoric Jul 19, 2022
7b094f3
WIP
Yoric Jul 19, 2022
fb54799
WIP
Yoric Jul 19, 2022
3cb4ffc
WIP
Yoric Jul 19, 2022
da4edb8
WIP
Yoric Jul 19, 2022
0cc6862
WIP
Yoric Jul 22, 2022
a9d3c2b
WIP
Yoric Jul 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
WIP
  • Loading branch information
@Yoric
Yoric committed Jul 19, 2022
commit da4edb88548fcce1405eb2542bfb4dadc5fc1839
4 changes: 2 additions & 2 deletions src/commands/CommandHandler.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,14 +48,14 @@ export const COMMAND_PREFIX = "!mjolnir";
export async function handleCommand(roomId: string, event: { content: { body: string } }, mjolnir: Mjolnir) {
const line = event['content']['body'];
const parts = line.trim().split(' ').filter(p => p.trim().length > 0);
console.debug("YORIC", "line", line);

const lexer = new Lexer(line);
lexer.token("command"); // Consume `!mjolnir`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what happens when the lexer fails then? how different is it from what happens now?

// Extract command.
const cmd = lexer.alternatives(
() => lexer.token("id").text,
() => null
);
console.debug("YORIC", "cmd", cmd);

try {
if (cmd === null || cmd === 'status') {
Expand Down
78 changes: 42 additions & 36 deletions src/commands/Lexer.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,92 +6,98 @@ import * as TokenizrModule from "tokenizr";
import { parseDuration } from "../utils";
const TokenizrClass = Tokenizr || TokenizrModule;

const WHITESPACE = /\s+/;
const COMMAND = /![a-zA-Z_]+/;
Yoric marked this conversation as resolved.
Show resolved Hide resolved
const IDENTIFIER = /[a-zA-Z_]+/;
const USER_ID = /@[a-zA-Z0-9_.=\-/]+:\S+/;
const GLOB_USER_ID = /@[a-zA-Z0-9_.=\-?*/]+:\S+/;
const ROOM_ID = /![a-zA-Z0-9_.=\-/]+:\S+/;
const ROOM_ALIAS = /#[a-zA-Z0-9_.=\-/]+:\S+/;
const ROOM_ALIAS_OR_ID = /[#!][a-zA-Z0-9_.=\-/]+:\S+/;
const INT = /[+-]?[0-9]+/;
const STRING = /"((?:\\"|[^\r\n])*)"/;
const DATE_OR_DURATION = /(?:"([^"]+)")|([^"]\S+)/;
const STAR = /\*/;
const ETC = /.*$/;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why keep these separate from the lexer rules?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kinda expect that we're going to need some of these regexps in other places, but I may be wrong.


/**
* A lexer for common cases.
* A lexer for command parsing.
*
* Recommended use is `lexer.token("state")`.
*/
export class Lexer extends TokenizrClass {
constructor(string: string) {
super();
console.debug("YORIC", "Lexer", 0);
// Ignore whitespace.
this.rule(/\s+/, (ctx) => {
this.rule(WHITESPACE, (ctx) => {
ctx.ignore()
})

// Command rules, e.g. `!mjolnir`
this.rule("command", /![a-zA-Z_]+/, (ctx) => {
this.rule("command", COMMAND, (ctx) => {
ctx.accept("command");
});

// Identifier rules, used e.g. for subcommands `get`, `set` ...
Yoric marked this conversation as resolved.
Show resolved Hide resolved
this.rule("id", /[a-zA-Z_]+/, (ctx) => {
this.rule("id", IDENTIFIER, (ctx) => {
ctx.accept("id");
});

// Users
this.rule("userID", /@[a-zA-Z0-9_.=\-/]+:.+/, (ctx) => {
this.rule("userID", USER_ID, (ctx) => {
ctx.accept("userID");
});
this.rule("globUserID", /@[a-zA-Z0-9_.=\-?*/]+:.+/, (ctx) => {
this.rule("globUserID", GLOB_USER_ID, (ctx) => {
ctx.accept("globUserID");
});

// Rooms
this.rule("roomID", /![a-zA-Z0-9_.=\-/]+:.+/, (ctx) => {
this.rule("roomID", ROOM_ID, (ctx) => {
ctx.accept("roomID");
});
this.rule("roomAlias", /#[a-zA-Z0-9_.=\-/]+:.+/, (ctx) => {
this.rule("roomAlias", ROOM_ALIAS, (ctx) => {
ctx.accept("roomAlias");
});
this.rule("roomAliasOrID", /[#!][a-zA-Z0-9_.=\-/]+:.+/, (ctx) => {
this.rule("roomAliasOrID", ROOM_ALIAS_OR_ID, (ctx) => {
ctx.accept("roomAliasOrID");
});

// Numbers.
this.rule("int", /[+-]?[0-9]+/, (ctx, match) => {
this.rule("int", INT, (ctx, match) => {
ctx.accept("int", parseInt(match[0]))
});

// Quoted strings.
this.rule("string", /"((?:\\"|[^\r\n])*)"/, (ctx, match) => {
this.rule("string", STRING, (ctx, match) => {
ctx.accept("string", match[1].replace(/\\"/g, "\""))
});

// Dates and durations.
console.debug("YORIC", "Lexer", 1);
try {
this.rule("dateOrDuration", /(?:"([^"]+)")|(\S+)/, (ctx, match) => {
let content = match[1] || match[2];
console.debug("YORIC", "Lexer", "dateOrDuration", content);
let date = new Date(content);
console.debug("YORIC", "Lexer", "dateOrDuration", "date", date);
if (!date || Number.isNaN(date.getDate())) {
let duration = parseDuration(content);
console.debug("YORIC", "Lexer", "dateOrDuration", "duration", duration);
if (!duration || Number.isNaN(duration)) {
ctx.reject();
} else {
ctx.accept("duration", duration);
}
this.rule("dateOrDuration", DATE_OR_DURATION, (ctx, match) => {
let content = match[1] || match[2];
let date = new Date(content);
if (!date || Number.isNaN(date.getDate())) {
let duration = parseDuration(content);
if (!duration || Number.isNaN(duration)) {
ctx.reject();
} else {
ctx.accept("date", date);
ctx.accept("duration", duration);
}
});
} catch (ex) {
console.error("YORIC", ex);
}
} else {
ctx.accept("date", date);
}
});

// Jokers.
this.rule("STAR", /\*/, (ctx) => {
this.rule("STAR", STAR, (ctx) => {
ctx.accept("STAR");
});

// Everything left in the string.
this.rule("ETC", /.*/, (ctx) => {
ctx.accept("ETC")
this.rule("ETC", ETC, (ctx, match) => {
ctx.accept("ETC", match[0].trim());
});

console.debug("YORIC", "Preparing lexer", string);
this.input(string);
}

Expand Down
33 changes: 14 additions & 19 deletions src/commands/SinceCommand.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,6 @@ function formatResult(action: string, targetRoomId: string, recentJoins: Join[],
async function execSinceCommandAux(destinationRoomId: string, event: any, mjolnir: Mjolnir, lexer: Lexer): Promise<Result<undefined>> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rather than passing the lexer to commands, wouldn't it be better if commands declared what arguments they are expecting (can be part of the same api to register with the handler)? This means that they don't need to know implementation detail of the lexer

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to move towards what you suggest. I'm not sure I'll succeed in a single pass, though.

// Attempt to parse `<date/duration>` as a date or duration.
let dateOrDuration: Date |number = lexer.token("dateOrDuration").value;
console.debug("YORIC", "dateOrDuration", dateOrDuration);
let minDate;
let maxAgeMS;
if (dateOrDuration instanceof Date) {
Expand All @@ -101,25 +100,28 @@ async function execSinceCommandAux(destinationRoomId: string, event: any, mjolni
if (!action) {
return {error: `Invalid <action>. Expected one of ${JSON.stringify(Action)}`};
}
console.debug("YORIC", "action", action);

// Attempt to parse `<limit>` as a number.
const maxEntries = lexer.token("int").value as number;
console.debug("YORIC", "maxEntries", maxEntries);

// Now list affected rooms.
// Parse rooms.
// Parse everything else as `<reason>`, stripping quotes if any have been added.
const rooms: Set</* room id */string> = new Set();
let reason = "";
do {

let token = lexer.alternatives(
// Room
() => lexer.token("STAR"),
() => lexer.token("roomAliasOrID"),
// Reason
() => lexer.token("string"),
() => lexer.token("ETC")
);
console.debug("YORIC", "token", token);
if (!token) {
// We have reached the end of rooms.
if (!token || token.type === "EOF") {
// We have reached the end of rooms, no reason.
break;
}
if (token.type === "STAR") {
} else if (token.type === "STAR") {
for (let roomId of Object.keys(mjolnir.protectedRooms)) {
rooms.add(roomId);
}
Expand All @@ -131,8 +133,9 @@ async function execSinceCommandAux(destinationRoomId: string, event: any, mjolni
}
rooms.add(roomId);
continue;
}
if (token.type == 'EOF') {
} else if (token.type === "string" || token.type === "ETC") {
// We have reached the end of rooms with a reason.
reason = token.text;
break;
}
} while(true);
Expand All @@ -141,14 +144,6 @@ async function execSinceCommandAux(destinationRoomId: string, event: any, mjolni
error: "Missing rooms. Use `*` if you wish to apply to every protected room.",
};
}
console.debug("YORIC", "rooms", rooms);

// Parse everything else as `<reason>`, stripping quotes if any have been added.
const reason = lexer.alternatives(
() => lexer.token("string"),
() => lexer.token("ETC")
)?.text || "";
console.debug("YORIC", "reason", reason);

const progressEventId = await mjolnir.client.unstableApis.addReactionToEvent(destinationRoomId, event['event_id'], '⏳');

Expand Down
2 changes: 1 addition & 1 deletion test/integration/fixtures.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ export const mochaHooks = {
console.error("---- entering test", JSON.stringify(this.currentTest.title)); // Makes MatrixClient error logs a bit easier to parse.
console.log("mochaHooks.beforeEach");
// Sometimes it takes a little longer to register users.
this.timeout(10000)
this.timeout(20000)
this.managementRoomAlias = config.managementRoom;
this.mjolnir = await makeMjolnir();
config.RUNTIME.client = this.mjolnir.client;
Expand Down