[Lua] Highlight strings used by string.{format,gsub,pack,etc} by mbartlett21 · Pull Request #4194 · sublimehq/Packages

mbartlett21 · 2025-03-20T13:28:09Z

This PR adds highlighting to the strings that are passed to string.{format,find,match,gsub,gmatch,pack,unpack,packsize}.

They are split into three categories:

Format strings
https://www.lua.org/manual/5.4/manual.html#pdf-string.format
These are done similarly to C's sprintf and friends, except that Lua
only supports some specifiers and adds q as an option.
Patterns
https://www.lua.org/manual/5.4/manual.html#6.4.1
They are passed as the second argument to the four functions that take
patterns, meaning that I can also match on
(whatever_value):match 'this%s*that'. For scopes, I have roughly
followed what the builtin regular expressions package does.
I haven't worked on nested groups as that got quite hard with handling
it inside the three different types of strings that Lua provides.
The special capture () returns a position. I'm not sure what scope
this should be. I've set it to punctuation.section.brackets.lua for
the moment.
Pack format strings
https://www.lua.org/manual/5.4/manual.html#6.4.2
The alignment and endian specifiers I have scoped as
storage.modifier and I have scoped the actual type designations as
storage.type. For the padding that can be specified with x or X,
I've scoped it to punctuation.separator.padding as I didn't see any
better one, and that helped differentiate it from ones that have
values.

Since format strings are always the first argument, I did not find a way
of handling them when the function is called as a method.
(e.g. ('%3.5f'):format(3)).

For format strings and pack strings, they now use branching inside
parenthesized-expression to choose what gets formatted.

jrappen · 2025-03-20T14:23:27Z

From a quick glance over this PR I noticed:

Changes to the syntax file are not completely covered by the added tests in the test file.
Changes suggest anchor keywords only match at beginning and end of single-line strings, but anywhere in multi-line strings. Maybe double-check that, I don't know Lua.

mbartlett21 · 2025-03-20T14:46:38Z

From a quick glance over this PR I noticed:

Changes to the syntax file are not completely covered by the added tests in the test file.

Noted. I'll probably look at the tests from eregex etc.

Changes suggest anchor keywords only match at beginning and end of single-line strings, but anywhere in multi-line strings. Maybe double-check that, I don't know Lua.

That is correct. Anchors are only matched at the ends of strings. elsewhere they match the character.
Edit: actually you're right. I'll need to check that

michaelblyons · 2025-03-20T15:15:05Z

It has become the practice more and more to use named contexts to push and set into, rather than successively indenting anonymous contexts. This lets inheriting syntaxes add extra rules in more locations without replicating big chunks of the base syntax.

mbartlett21 · 2025-03-21T01:12:46Z

@jrappen

I've added (roughly) the tests from the Regular Expression package and worked on getting the syntax to work with character classes and captures now.

Changes suggest anchor keywords only match at beginning and end of single-line strings, but anywhere in multi-line strings. Maybe double-check that, I don't know Lua.

This is fixed now so that the anchor only matches at the very start and very end. (Because of how multiline strings work, it has to account for a possible preceding newline)

@michaelblyons
Were you meaning the sections under pattern-{single-quoted,double-quoted,multiline}-string-body? I don't think there is any nested ones left in what's been added now.

Lua/Lua.sublime-syntax

mbartlett21 · 2025-03-24T11:09:38Z

I've added some lookahead matching on the one line for immediate calls so that it can work with ("string"):{pack,format}.

Lua/Lua.sublime-syntax

jrappen · 2025-03-25T22:05:08Z

remove the todo list from the opening comment when done
find a way to get rid of the anonymous contexts five (?!) levels deep

mbartlett21 · 2025-03-25T22:35:23Z

find a way to get rid of the anonymous contexts five (?!) levels deep

This seems to be to do with with_prototype, which I've used in order to deduplicate code between double-quoted and single-quoted strings.

Or are you meaning in the builtin-modules pattern structure?

michaelblyons · 2025-03-25T23:30:15Z

Instead of something like

strings:
  - match: '"'
    scope: punctuation.definition.string.begin
    push:
      - meta_scope: string.quoted.double
      - match: '"'
        scope: punctuation.definition.string.end
        pop: 1

Use

strings:
  - match: '"'
    scope: punctuation.definition.string.begin
    push: string-double-body

string-double-body:
  - meta_scope: string.quoted.double
  - match: '"'
    scope: punctuation.definition.string.end
    pop: 1

- add named contexts - add `leading_wspace` var - add `first_line_language` var

Lua/Lua.sublime-syntax

deathaxe · 2025-05-17T18:49:20Z

Lua/Lua.sublime-syntax

+        - match: (?="(?:{{string_char_escape}}|[^\\"])*"\s*\)\s*:\s*format{{identifier_break}})
+          push: fmtstr-expression
+        - match: (?='(?:{{string_char_escape}}|[^\\'])*'\s*\)\s*:\s*format{{identifier_break}})
+          push: fmtstr-expression
+        - match: (?="(?:{{string_char_escape}}|[^\\"])*"\s*\)\s*:\s*(?:pack|unpack|packsize){{identifier_break}})
+          push: packstr-expression
+        - match: (?='(?:{{string_char_escape}}|[^\\'])*'\s*\)\s*:\s*(?:pack|unpack|packsize){{identifier_break}})
+          push: packstr-expression


Those patterns expect pack/format strings to be one-liners.

But even expressions of concatted strings (e.g.: ("one %0s two" + "another %d"):format();) are not scoped format string.

Other syntax definitions like python, don't restrict placeholder this way, as it is also valid to specify a format string in a separate variable.

fmt_str = "My %s format" fmt_str:format("name")

I'd suggest to highlight placeholders everywhere or head for a more flexible approach using branching to scope all strings in a group followed by :format(), ... accordingly.

It should now be working. The only thing I don't allow is having the : and format on separate lines. (which is technically valid but is a weird style)

I also updated the other expressions to allow things like string.format("%s " .. "%s") which they didn't before (only the first string was highlighted)

jrappen · 2025-07-07T11:50:11Z

@mbartlett21 maybe you could address the change-requests?

It now allows multiple strings to be highlighted. For example string.match("this", "%w".."hi".."%S")

mbartlett21 · 2026-01-06T00:28:00Z

@deathaxe / @jrappen

I've finally gotten around to doing this. Will I need to rebase to get CI to work or is there no dev build at the moment?

jrappen · 2026-01-06T00:30:30Z

SublimeHQ were doing some infrastructure updates for their linux builds over the last couple of days and re-built the last dev and stable builds. I guess something broke?

/cc @BenjaminSchaaf

jrappen · 2026-01-06T04:56:10Z

Ben said he fixed their side.

jrappen · 2026-01-12T10:53:36Z

Lua/Lua.sublime-syntax

+      scope: invalid.illegal.unclosed-string.lua
+      pop: 2
+    - match: \\.
+      scope: invalid.illegal.invalid-escape.lua


this should've been invalid.illegal.escape-character.lua

jrappen · 2026-01-12T10:55:36Z

Lua/Lua.sublime-syntax

  parenthesized-expression:
    - match: \(
      scope: punctuation.section.group.begin.lua
+      branch_point: special-string-paren


It might make sense to name branch point the same as the context containing it.

jrappen · 2026-01-12T10:58:02Z

Lua/Lua.sublime-syntax

+        - parenthesized-expression-end-normal
+        - parenthesized-expression-inside-normal
+
+  # at this point the stack is:


Maybe remove these comments as well as the "pop this as well as ..." ones below? They were mainly for yourself while you were working on this. We don't need them.

jrappen · 2026-01-12T11:03:31Z

Lua/Lua.sublime-syntax

+  pattern-unquoted-charclass:
+    - meta_include_prototype: false
+    - include: pattern-shared-charclass
+    - match: \%b.?.?


\%b.{0,2} might be more explicit

jrappen · 2026-01-12T11:24:50Z

I didn't have a lot of time to look through this.

you might want to review tests where meta.* scopes are used so that:

meta.mapping vs. meta.mapping.key vs. meta.mapping.value are correctly tested like

-- SYNTAX TEST "Packages/Lua/Lua.sublime-syntax"

    KEY SEPARATOR VALUE
--  ^^^^^^^^^^^^^^^^^^^ meta.mapping
--  ^^^ meta.mapping.key
--     ^ - meta.mapping.key
--               ^^^^^ meta.mapping.value
--                     ^ - meta.mapping

also check strings are correctly tested, especially multiline ones.
Use character combinations for the beginning and the end of test lines using the definition in the first line, compare:
```
# SYNTAX TEST "Packages/x/x.sublime-syntax \

multilinestring = "whatever \
#                           ^ separator.line \
"
```
unless lua multiline string work with """ whatever """ then you'd not need to define line separators like above. But you'll figure it out.
the structure of the context for multiline strings looks different from other languages, I didn't have too much time to look through those. If not everything is working as intended, maybe check other language syntaxes with multiline strings.
some added changes to the syntax file are not reflected in the tests added

mbartlett21 added 4 commits March 20, 2025 22:56

Special case pattern strings

bb6572d

Special case format strings

8ace4cf

Special case pack strings

b310c1a

Add test

c6974f0

mbartlett21 added 8 commits March 21, 2025 09:27

Further work on patterns

e3caf59

More work on patterns

a875bb8

More work on Lua patterns

933aa66

Add test for string.format

4659e1d

Remove unused match

ed8a400

Fix escapes

61c450f

Change scope

1215617

Add extra test

2bf42bb

deathaxe reviewed Mar 21, 2025

View reviewed changes