Jump to content
laes

IDE Syntax Highlighter using Tree-sitter

Recommended Posts

Hi all,

 

I've always missed a modern implementation of a syntax highlighting in RAD Studio. So when I saw a github repo containing pascal grammar for Tree-sitter, I started working on an IDE extension that makes use of it.

Since there are no Delphi bindings for Tree-sitter I wrote a simple TCP Server in Rust, which receives the currently opened source code, parses it and generates the highlight information. After this information is sent back to the IDE I use it to draw more informative highlighting. Simple example below.

Untitled.thumb.png.cfb560bc1b88e2a583547d377e9862a3.png

 

I'm not sure about the performance, since it sends the code, parses and sends back the entire ToolsApi.pas file with 83k lines in 421 milliseconds. But it doesn't need to reparse the entire file on changes so it should be able to provide real-time highlighting, although I haven't implemented that yet.

Now the grammar isn't finished and I haven't really looked into how to write grammar files, plus I really don't know enough about the Pascal and Delphi language to undertake something like that.

 

The Point of this post is to gauge if there is even any interest for an extension like this. 

  • Like 5

Share this post


Link to post
11 hours ago, Edwin Yip said:

Hi @laes, thanks for your efforts! In case you didn't know, the CN Pack Delphi extension provides some advanced syntax highlighting: http://www.cnpack.org/images/cnwizards.gif

I do know about CnPack and I've used it before but I don't think it supports the kind of highlighting based on symbol types that tree-sitter generates. As an aside I would like to make my highlighting work alongside extensions like CnPack but right now the drawing performance goes out the window if I have CnPack running as well. If someone can point me to an example where this has been done I would very much appreciate it.

Share this post


Link to post

Hey @laes , did you had up publishing this work ? I'd love to have it, working without syntax highlighting is hell.

Share this post


Link to post

So, from your description I gather you are not using the LSP? Is something missing from it to make it unsuitable for this task, or is it not accessible altogether for plugins other than Embarcaderos own? Or am I misunderstanding the purpose of the LSP completely...never actively used it.

Share this post


Link to post
Just now, Sherlock said:

is it not accessible altogether for plugins other than Embarcaderos own?

The LSP protocol is documented, but one has to implement the complete client code from the ground up and that is a pretty complex task. Currently there is no ready to use API provided by the IDE.

  • Thanks 1

Share this post


Link to post

@Uwe Raabe Thanks! Sounds like a tough task then. But if an API existed, would it make this task easier? I'm still trying to figure out, where one could benefit from the LSP other than finally getting correctly marked errors in the source.

Share this post


Link to post
1 hour ago, Sherlock said:

But if an API existed, would it make this task easier?

If such an API would wrap things like "provide LSP with all necessary code for the current project" and "notify this interface with the requested results" that would make things easier. Even not trivial, though.

Share this post


Link to post

I have not yet looked into the project, but there is this:
https://github.com/rickard67/LSP-Pascal-Library

and this:
https://github.com/rickard67/language-server-protocol

They were written, it seems, in support of a text editor by the same developer. He provides an unusually large amount of info with snippets demonstrating calls. Seems very helpful, but it also makes me think the task is challenging, even with what he has already written.

Edited by Bill Meyer
  • Like 1
  • Thanks 2

Share this post


Link to post
On 4/18/2022 at 8:23 AM, laes said:

gauge if there is even any interest for an extension like this. 

Answer: Yes. This is awesome.

Did you ever publish it anywhere?

I have lots of questions - things like, are you using the editor painting API to paint?

This is my personal DelphiPraxis account, but you could email me at david.millington@embarcadero.com -- I'd love to chat about it.

Share this post


Link to post
On 4/18/2022 at 3:23 AM, laes said:

Since there are no Delphi bindings for Tree-sitter

Since your post actually made me start doing a Delphi binding, allow me a shameless plug: https://github.com/modersohn/delphi-tree-sitter

 

I'm still busy covering queries, but it should be a matter of days before I've covered these as well.

 

And yeah - I too would very much love to see how you did the querying. And I've noticed that tree-sitter-pascal in its current form could not handle some sources I threw at it - but I didn't yet actually figure out what caused this, too busy getting the basics done.

Share this post


Link to post
7 hours ago, msohn said:

Since your post actually made me start doing a Delphi binding,

I've already made a full binding of the APIs. It isn't published yet, as it is part of my formatter and syntax highlighter product I'm building for Delphi (and C++Builder)

 

 

On 3/16/2024 at 3:55 PM, Dave Millington (personal) said:

I have lots of questions - things like, are you using the editor painting API to paint?

The current editor highlighter toolsapi interface is limited.
To get the full benefits of the syntax highlighting data tree-sitter can deliver, complete takeover of the editor painting is necessary.

 

7 hours ago, msohn said:

And I've noticed that tree-sitter-pascal in its current form could not handle some sources I threw at it

The grammar for the pascal parser needs an update indeed.

  • Like 2

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×