While this seems like a nice effort the roadmap for Neovim has Tree-sitter¹ integration² planned for the next release (0.5).

Tree-sitter - A New Parsing System For Programming Tools

"Developer tools that support multiple programming languages generally have very limited, regex-based code-analysis capabilities.

Tree-sitter is a new parsing system that aims to change this paradigm. It provides a uniform C API for parsing an ever-growing set of languages. It features high-performance incremental parsing and robust error recovery, which allow it to be used to parse code in real-time in a text editor.

We're in the process of integrating Tree-sitter into both GitHub.com and the Atom text editor, which will allow us to analyze code accurately and efficiently, paving the way for better syntax highlighting, code navigation, and refactoring." -- Here's an interesting talk about it: https://www.thestrangeloop.com/2018/tree-sitter---a-new-pars...

¹ http://tree-sitter.github.io/tree-sitter/

² https://github.com/neovim/neovim/pull/10124

I mean, to be fair, this only took me a few hours to write, and most of it was spent on the README/docs/demo, so it's not a huge deal. I do wish I had done it sooner, considering how long the Lua integration has been in Neovim. Even the trie implementation was rather easy since I used a naive algorithm (but it was fun to flex my ffi).

I think treesitter would be overkill for this, however, considering the size and ffi overhead involved and any speed improvement would be very minimal while increasing the complexity. Considering that we are only told what "lines" changed by neovim right now, it would be very difficult to improve on what I have right now.

I had seen mention of treesitter in the code, and I had no idea what it was, so thank you very much for enlightening me. I have a decent amount of experience writing pratt parsers and DFAs by hand, which will probably still be faster, but the incremental algorithm and other advanced features wouldn't come for free.

I had been looking for more advanced reading material on parsers (I was even thinking of emailing Donald Knuth to see if I could preview chapter 7's section on parsing), and also for parsers in some of those languages, so this is very nicely timed. I'm definitely going to be reading their papers and watching the talk.

>While this seems like a nice effort the roadmap for Neovim has Tree-sitter¹ integration² planned for the next release

Well, one is an existing syntax highlighter one can install and use now, the other is something new, we haven't seen, that might or might not come with the next release.

Does anybody use it atm on any editor?

Tree sitter is used in Atom and is replacing Textmate grammars. In fact, tree sitter recently got s-expression query support, which will replace the old CSS selector syntax. So atom is already in the second generation of tree sitter support.

GitHub semantic (go to definition, call graph) is powered by tree sitter wrapped in an Haskell analyzer.

The next version of Codemirror is using Lezer, a JS port of tree sitter.

I use tree sitter in a static analysis tool that runs on our internal repos (about 4000).

Tree sitter? It can be used with an extension in VS Code:

https://marketplace.visualstudio.com/items?itemName=georgewf...

It works just fine, although it is not a massive improvement by any means, from the dev experience we have today.

I do not know if any editors are shipping with it yet.

Presumably also Atom and GitHub's online editor, for which Tree-sitter was written.

Oh! This looks great. Regex parser can have nasty side effects.

But for C++ it still seems like something libclang based is still better right?

Edit: I see there is a discussion on asynchronous highlighting work, what's the status on that?

https://github.com/jeaye/color_coded

Ahh, this is why I love Neovim: plugins in virtually any language.

Good old vim has bindings in many languages, too. I don’t think this is a neovim selling point?

iirc Neovim has been trying to pretty much totally supplant Vimscript with Lua

Which is a good thing.

Better to use a more established language than reinventing the wheel

One thing to consider is that Lua is not a programming language that honors the seamless evolution. Essentially there are three flavors of Lua identified by their versions (5.1, 5.2 and 5.3), almost identical to each other but different enough that a migration is unreasonable. If it were not the case, there is no reason that NeoVim uses Lua 5.1 [1] even though 5.3 is the latest and 5.4 beta is on the way [2]. I think this dependency on 5.1 can become a huge problem later.

[1] https://neovim.io/doc/user/if_lua.html

[2] http://lua-users.org/lists/lua-l/2019-10/msg00003.html

NeoVim doesn't care about Lua, they care about LuaJit, which doesn't (and probably wont ever) support lua syntax > 5.1 (or 5.2).

>almost identical to each other but different enough that a migration is unreasonable.

There's nothing unreasonable, and all versions of Lua mentioned are backwards compatible, so there's not even a problem about migrating. Going to a new version is as simple as installing it and running your program.

> Going to a new version is as simple as installing it and running your program.

Not true. Each minor version of Lua tended to inject incompatibilities that can't be trivially resolved. Lua 5.2 had made a catastrophic change to the environment [1], and Lua 5.3 was less severe but there are subtle issues with integer-number conversions you won't ever want in your large codebase [2]. And that is not a theoretical matter, we had to stick to Lua 5.1 exactly due to the environment changes even though 64-bit integers would have made our codebase much cleaner. And Lua 5.1 and 5.2 are no longer supported [3] (though as you have said, LuaJit may be supported) despite of all remaining issues.

[1] https://www.lua.org/manual/5.2/manual.html#8

[2] https://www.lua.org/manual/5.3/manual.html#8

[3] https://www.lua.org/versions.html

True, the environment change was a big one.

But otherwise many codebases do move up Lua versions (outside of LuaJit), it's not a Python 3 "wait 10 years", Perl 6 "Wait 20 years" situation/

Python 2.7 was supported for a decade, providing a reasonable environment when Python 3 was not really matured enough. And while it didn't work, Python 3 tried to provide the migration path.

Perl 6 requires the full p5 compatibility as far as I recall. It was designed to coexist.

Lua only provides some C preprocessor flags for partial backward compatibility.

I suspect the reason they stick to 5.1 is because that's what LuaJIT supports.

LuaJIT basically dead, it's like if PyPy would forever stuck just to Python 2 support.

Without Mike Pall, it won't be progressing as quickly, but it's definitely not dead. It can still fill a niche as a best in class embedded language thanks to its amazing JIT. It's used in Redis, mpv, neovim, and a lot of other very popular software projects.

However, I'm hoping to learn more about the internals of the JIT to see if I can take up some of the effort in progressing it. I have a very deep interest in Luajit for some reason. As long as people like me exist, it's not dead.

Not dead, just mature. Also has a fork and it's used in other projects.

https://github.com/raptorjit/raptorjit

Which makes it a very good pick for NeoVim. It basically means that the plugins are basically supported forever without the JIT being a moving target. NeoVim also supports Vimscript as an alternative.

As I understand it, they want to implement vimscript in Lua, not (necessarily) elevate Lua to be a first class scripting language next to vimscript.

Lua already is a first class, built in scripting language in Neovim. (it can be disabled at build time)

I think adoption will remain pretty limited though, because plugin authors want to support vim and neo.

> Lua already is a first class, built in scripting language in Neovim.

Yes.

> (it can be disabled at build time)

No, Lua is not optional in Neovim.

Related with this: does anyone know of a good Lua indent file for Vim? All the ones I have tried have trouble with indenting things (writing `busted` specs in particular tends to be really annoying, the space on the next line after the `describe` is never right).