> In simpler words, as a user: The jq query language [1] is obtuse, obscure, incredibly hard to learn if you need it for quick one liners once in a blue moon.
I don't agree that jq's query language is obtuse. It's a DSL for JSON document trees, and it's largely unfamiliar, but so is xpath or any other DOM transformation language.
The same thing is said about regex.
My take is that "it's obtuse" just translates to "I'm not familiar with it and I never bothered to get acquainted with it".
One thing that we can agree though is that jq's docs are awful at providing a decent tutorial for new users to ramp up.
I'm pretty good with regular expressions. I have spent a lot of time trying to get familiar with jq. The problem is that I never use it outside of parsing JSON files, yet I use regular expressions all over the place: on the command line, in Python and Javascript and Java code. They are widely applicable. Their syntax is terse, but relatively small.
jq has never come naturally. Every time I try to intuit how to do something, my intuition fails. This is despite having read its man page a dozen times or more, and consulted it even more frequently than that.
I've spent 20+ years on the Unix command line. I know my way around most of it. I can use sed and awk and perl to great effect. But I just can't seem to get jq to stick.
Aside, but there's a lot of times when "I know jq can do this, but I forget exactly how, let me find it in the man page" and then... I find jq's man page as difficult as jq itself when trying to use it as a reference.
Anyway, $0.02.
Edited to add: as a basic query language, I find it easy to use. It's when I'm dealing with json that embeds literal json strings that need to be parsed as json a second time, or when I'm trying to manipulate one or more fields in some way before outputting that I struggle. So it's when I'm trying to compose filters and functions inside jq that I find it hard to use.
Agreed. I only use jq once a month or once in two months at most. Every time I want to do something I just search for my use case since I can't seem to remember the syntax.
But I have tried to learn jq's syntax (it's pretty much a minilanguage) and it has been incredibly difficult.
I also remember what when I first tried learning regex it was also very difficult. That is until I learned about finite state machines and regular languages, after that CS fundamentals class I was able to make sense of regex in a way that stuck.
Is there a comparable theory for jq's mini-language?
Not a theory per se, but my "lightbulb moment" with jq came when I thought about it like this:
jq is basically a templating language, like Jsonnet or Jinja2. What jq calls a "filter" can also be called a template for the output format.
Like any template, a jq filter will have the same structure as the desired output, but may also include dynamically calculated (interpolated) data, which can be a selection from the input data.
So, at a high level, write your filter to look like your output, with hardcoded data. Then, replace the "dynamic" parts of the output data with selectors over the input.
Don't worry about any of the other features (e.g. conditionals, variables) until you need them to write your "selectors."
I don't know of any formal theory, but it feels a bit like functional programming because you don't often use variables (an advanced feature, as the manual says). I kind of got a feel for it by realizing that it wants to push a stream of objects through transformations, and that's about it. A few operators/functions can "split" the stream, or pull the stream back into place. Like, uh,
in.json {"a":1,"b":2}
jq -c '{a}' in.json
{"a":1}
The . is the current stream, so if I just do ". , .", it's kind of pushing two streams along:
jq -c '.,. | {a}' in.json
{"a":1}
{"a":1}
Then, of course, say:
jq -c '{a, b, c: .}' in.json
{"a":1,"b":2,"c":{"a":1,"b":2}}
It was going through the . stream, and I pulled the . stream right back in while doing so.
So it kind of helps to keep straight in my head when I've kind of got multiple streams going, vs multiple values.
Someone (almost anyone) can probably explain better with formal theory, but I just kind of got a feel for it and kind of describe it like this.
I think it's more like "I'm not familiar with it and getting it to do something that seems like it should be easy is surprisingly hard, even though I'm putting in some effort." I've become pretty good at jq lately, but for several years before that I would occasionally have some problem that I knew jq could solve, and resolved to sit down and learn the damn thing already, and every time, found it surprisingly difficult. Until you get a really good understanding of it (and good techniques for debugging your expressions), it's often easier just to write a python script.
I love jq, and without detracting from it, gron looks like an extremely useful, "less difficult" complement to it.
Adding: in fact, gron's simplicity is downright inspired. It looks like all it does is convert your json blob into a bunch of assignment statements that have the full path from root to node, and the ability to parse that back into an object. Not sure why I didn't think of that intermediate form being way more greppable. Kudos to the author.
Just as an example, this just took me about a minute to get the data I wanted, whereas I probably spent a half an hour on it yesterday with jq:
I don't agree that jq's query language is obtuse. It's a DSL for JSON document trees, and it's largely unfamiliar, but so is xpath or any other DOM transformation language.
The same thing is said about regex.
My take is that "it's obtuse" just translates to "I'm not familiar with it and I never bothered to get acquainted with it".
One thing that we can agree though is that jq's docs are awful at providing a decent tutorial for new users to ramp up.