IDE Syntax Highlighter using Tree-sitter

[laes 5]

Hi all,

 

I've always missed a modern implementation of a syntax highlighting in RAD Studio. So when I saw a github repo containing pascal grammar for Tree-sitter, I started working on an IDE extension that makes use of it.

Since there are no Delphi bindings for Tree-sitter I wrote a simple TCP Server in Rust, which receives the currently opened source code, parses it and generates the highlight information. After this information is sent back to the IDE I use it to draw more informative highlighting. Simple example below.

 

I'm not sure about the performance, since it sends the code, parses and sends back the entire ToolsApi.pas file with 83k lines in 421 milliseconds. But it doesn't need to reparse the entire file on changes so it should be able to provide real-time highlighting, although I haven't implemented that yet.

Now the grammar isn't finished and I haven't really looked into how to write grammar files, plus I really don't know enough about the Pascal and Delphi language to undertake something like that.

 

The Point of this post is to gauge if there is even any interest for an extension like this. 

[Edwin Yip 154]

Hi @laes, thanks for your efforts! In case you didn't know, the CN Pack Delphi extension provides some advanced syntax highlighting: http://www.cnpack.org/images/cnwizards.gif

[laes 5]

11 hours ago, Edwin Yip said:

Hi @laes, thanks for your efforts! In case you didn't know, the CN Pack Delphi extension provides some advanced syntax highlighting: http://www.cnpack.org/images/cnwizards.gif

I do know about CnPack and I've used it before but I don't think it supports the kind of highlighting based on symbol types that tree-sitter generates. As an aside I would like to make my highlighting work alongside extensions like CnPack but right now the drawing performance goes out the window if I have CnPack running as well. If someone can point me to an example where this has been done I would very much appreciate it.

[dormky 2]

Hey @laes , did you had up publishing this work ? I'd love to have it, working without syntax highlighting is hell.

[Sherlock 663]

So, from your description I gather you are not using the LSP? Is something missing from it to make it unsuitable for this task, or is it not accessible altogether for plugins other than Embarcaderos own? Or am I misunderstanding the purpose of the LSP completely...never actively used it.

[Uwe Raabe 2057]

Just now, Sherlock said:

is it not accessible altogether for plugins other than Embarcaderos own?

The LSP protocol is documented, but one has to implement the complete client code from the ground up and that is a pretty complex task. Currently there is no ready to use API provided by the IDE.

[Sherlock 663]

@Uwe Raabe Thanks! Sounds like a tough task then. But if an API existed, would it make this task easier? I'm still trying to figure out, where one could benefit from the LSP other than finally getting correctly marked errors in the source.

[Uwe Raabe 2057]

1 hour ago, Sherlock said:

But if an API existed, would it make this task easier?

If such an API would wrap things like "provide LSP with all necessary code for the current project" and "notify this interface with the requested results" that would make things easier. Even not trivial, though.

[Bill Meyer 337]

I have not yet looked into the project, but there is this:
https://github.com/rickard67/LSP-Pascal-Library

and this:
https://github.com/rickard67/language-server-protocol

They were written, it seems, in support of a text editor by the same developer. He provides an unusually large amount of info with snippets demonstrating calls. Seems very helpful, but it also makes me think the task is challenging, even with what he has already written.

Edited August 3, 2022 by Bill Meyer

[Dave Millington (personal) 42]

On 4/18/2022 at 8:23 AM, laes said:

gauge if there is even any interest for an extension like this. 

Answer: Yes. This is awesome.

Did you ever publish it anywhere?

I have lots of questions - things like, are you using the editor painting API to paint?

This is my personal DelphiPraxis account, but you could email me at david.millington@embarcadero.com -- I'd love to chat about it.

[msohn 28]

On 4/18/2022 at 3:23 AM, laes said:

Since there are no Delphi bindings for Tree-sitter

Since your post actually made me start doing a Delphi binding, allow me a shameless plug: https://github.com/modersohn/delphi-tree-sitter

 

I'm still busy covering queries, but it should be a matter of days before I've covered these as well.

 

And yeah - I too would very much love to see how you did the querying. And I've noticed that tree-sitter-pascal in its current form could not handle some sources I threw at it - but I didn't yet actually figure out what caused this, too busy getting the basics done.

[Glenn Dufke 17]

7 hours ago, msohn said:

Since your post actually made me start doing a Delphi binding,

I've already made a full binding of the APIs. It isn't published yet, as it is part of my formatter and syntax highlighter product I'm building for Delphi (and C++Builder)

 

 

On 3/16/2024 at 3:55 PM, Dave Millington (personal) said:

I have lots of questions - things like, are you using the editor painting API to paint?

The current editor highlighter toolsapi interface is limited.
To get the full benefits of the syntax highlighting data tree-sitter can deliver, complete takeover of the editor painting is necessary.

 

7 hours ago, msohn said:

And I've noticed that tree-sitter-pascal in its current form could not handle some sources I threw at it

The grammar for the pascal parser needs an update indeed.