Congratulations, good job! The chaos of notebooks needs some tracking indeed.
7 years ago I made a project with 100 calculation dependencies, (in Python & SQL scripts) and the only thing that allowed not to loose track was Makefile + GraphViz.
I wanted to make something similar in Rust -- a static visualized of dependencies between structs. Things turned out way harder than expected.
Thanks! Yes I think a caching solution like this is great for notebooks, because it makes it very cheap to re-run the whole thing (as long as you reasonably organized long-running computations into `@op`s), overcoming the notorious state problem.
Graphviz is indeed a lifesaver; and you can similarly think of `mandala` as a "build system for Python objects" (with all the cool and uncool things that come with that; serializing arbitrary Python objects with strong guarantees is hardhttps://amakelov.github.io/mandala/tutorials/gotchas/).
I've no experience with rust, but I'd be curious to hear about the difficulties that came up. I'd expect to see some overlap with Python!
1. Imports are more complex than in Python, because a module can be just a block in code, not necessarily a separate file/folder. E.g. `pub mod my_module { <code with functions and constants> }` is a module inside another module, so you don't need a folder and `__init__` inside to have inner modules.
Also, `use something` may man an external crate.
`use super::something` means import from upper level of modules tree, but it's not necessarily a folder.
2. I can parse what types my functions require in their signatures, or structs require in their members (but I must have resolved where really those names point at), but there's also type elision -- i.e. you don't need to explicitly write the type of every var, it's deduced from what is assigned, for example `let my_var = some_func(...)` -- will make `my_var` have the return type of some_func. Now I must also keep track of all functions and what they return.
And then, there are generics:
let my_var: Vec<MyType> = vec![...];
Vec is generic, and in this case it has MyType inside. Well, this may be enough to just register `MyType` on my deps list. But then I may do some more calls:
let my_other_var = my_var.pop().unwrap().my_method();
Here, `pop()` returns `Option<MyType>`, unwrap returns `MyType`, and then in the end, my_method may return whatever, and I essentially need something like a compiler to figure out what it returns.
This seems big like a little compiler or language server.
7 years ago I made a project with 100 calculation dependencies, (in Python & SQL scripts) and the only thing that allowed not to loose track was Makefile + GraphViz.
I wanted to make something similar in Rust -- a static visualized of dependencies between structs. Things turned out way harder than expected.