NIP-91: AND operator for filters (Formerly NIP-119)

[dskvr]

Filter operand that has optimization benefits for relays, users and developers. Likely contentious.

This NIP is the result of convergent efforts. I wrote this NIP because it seemed obvious but didn't submit because I didn't have an implementation, @v0l wrote an implementation(*) around the same time. @lez connected the dots.

Rendered NIP

Discourse

• issue

Implementations

• nostr-rs-relay docker image
• satellite-node
• netstr
• worker-relay
• applesauce
• CodyTseng/nostr-relay
• nostr-relay-nestjs
• nostr-relay-tray
• nostr-relay-sqlite
• amethyst
• nostr.land

pending

• *nostr-rs-relay (draft PR)
• strfry PR #1
• strfry PR #2
• nostr-tools

Rationale

• Reduce bandwidth for all, meme AND cat objectively consumes less bandwidth than meme OR cat
• Reduce clock-time for relays, indexing with AND is fast for all common index formats, and faster compared to OR for some index formats. (See section below)
• Reduce client-side caching requirements
• Reduce centralization vectors by reducing or even eliminating the need for centralized REST, GraphQL APIs or specialized relay "feed" endpoints.
• Give relays the option to be more useful at the protocol level while improving efficiency for all parties.

Considerations

• New field for NIP-11.limitations: max_tags_per_and and max_tags_and
• Benchmarking should be conducted to validate that bandwidth and protocol usability as benefits supercede implementation and clock-time cost.

Index Efficiency

Index Type	AND Operation Efficiency	OR Operation Efficiency	Notes
B-Tree	High	Moderate	B-Tree indexes are very efficient for AND operations, especially with compound indexes. For OR operations, they are less efficient than for AND, as the database engine might need to traverse multiple paths.
Bitmap	High	High	Bitmap indexes excel in both AND and OR operations, particularly for columns with low cardinality. They utilize fast bitwise operations, making them ideal for read-heavy environments.
Hash	Not Applicable	Not Applicable	Hash indexes are designed for equality checks and do not directly support range-based queries or optimize for AND/OR operations efficiently.
Full-Text	High	High	Optimized for text search, full-text indexes efficiently handle both AND and OR conditions, making them suitable for complex text queries.

[alexgleason]

Have you implemented a relay? I can't even make tag queries fast as they are.

[dskvr]

Not yet, no. But I've scraped, researched and monitored them for almost two years.

There is a PR to nostr-rs-relay linked above from @v0l that uses this syntax and it includes benchmarks

I can't even make tag queries fast as they are.

AND operators are (generally) faster for btree and (relatively) equivalent for all other indexing methods (depending on impl. ofc), so if you are using btree indices AND operators via filters are likely faster (depending on case ofc).

[hzrd149]

Satellite Node relay now supports AND tag filters satellite-earth/core@15b8132
Although no one is running it yet 😁

[mikedilger]

I cannot think of a use case where I want to ask a relay for only events that have the same tag twice with two different values.

[dskvr]

@mikedilger @fiatjaf Really? I can think of an infinite number of cases.

Here's one: { kinds: [2003], "&t": [ "movie", "4k" ] } Re: NIP-35 re: @v0l

To retrieve that combination in a populated dataset client-side could easily be <10% of the result provided by #t. However, this isn't the best example because most events with t,4k will be a movie.

And another: { kinds: [2003], "&t": [ "movie", "4k", "turkish" ] }

easily <0.1%.

[fiatjaf]

I think this proposal is reasonable if we limit & to 2 values -- more than that and it becomes infeasible for filtering on the relay side.

But even if it's reasonable I also think it's a slippery slope and soon people will be demanding full SQL support on relays.

[alexgleason]

Postgres has a @> operator that is almost identical in functionality to Nostr filters, except it uses AND instead of OR for array values. If anything supporting OR was actually harder, because I had to break the query up into multiple @> statements for each value in the list.

We have been able to work around Nostr filter limitations by changing the events themselves in many cases. I think the torrent category example (or Amazon product filtering as another hypothetical), is still basically unsolved, and we worked around that by simply not supporting the ability to do it. 😃

I'm not sure it's worth changing NIP-01 filters. I've been shoehorning any extra functionality I need into NIP-50 search extensions.

[dskvr]

I think this proposal is reasonable if we limit & to 2 values

Why not just leave it to the operator to decide? Maybe i run a relay that stores a unique event kind and I have optimized it to support up to 20 and filters. Why shouldn't I be able to allow a client to run REQs up to my determined limit?

But even if it's reasonable I also think it's a slippery slope and soon people will be demanding full SQL support on relays.

Possible, and fair point. However, as proposed it's an optimization that just so happens to be a feature. If data is provided to prove that there is optimization benefits for all parties (relays, clients, users) this would establish a baseline that could be used as a measure against filter creep.

I'm not sure it's worth changing NIP-01 filters.

As proposed it's additive and does not "change" NIP-01 filters.

[vitorpamplona]

Can we consider this to be part of NIP-01?

I really dislike having to check if each relay offers custom commands before even enabling the UI for the feature in the client. Maybe that's going to be normal in the future, but we should consider minimizing the array of optional base-level stuff available to clients. Or maybe we go all out with tons of customizations and make checks for custom filter features the norm and in that case, something like NIP-11 must be merged with NIP-01.

[bezysoftware]

How would current relay implementations signal supporting this if it becomes part of 01? I think making this a separate NIP is better, with NIP 11 signaling. Much like NIP 70, that feels pretty similar.

And perhaps make NIP 11 required.

[vitorpamplona]

How would current relay implementations signal supporting this if it becomes part of 01?

It would just be required for all relays to implement it. There is no signaling needed.

My issue is that we are starting to have all these additional base protocol features without having a good signaling mechanism that must be present and accurate in all relays (most NIP-11 documents do not reflect what is implemented in the relay, so NIP-11 is quite useless for now)

[bezysoftware]

My question was more about the transition period after merging. The current relay implementations wouldn't magically start supporting this (let alone running instances), you would need a mechanism to check for support.

I agree the current state isn't great, but frankly that's always going to be a problem. You'll always have to code defensively and have fallbacks when a relay says one thing (or nothing), but behaves differently.

[vitorpamplona]

Agree. We can have:

1. A minimum required set of features for the base protocol + poor signaling support and a few very rare add-ons
2. An expanded required set of features for the base protocol + poor signaling support and a few very rare add-ons
3. A minimum required set of features for the base protocol + great signaling support and great add-ons
4. An expanded required set of features for the base protocol + great signaling support and great add-ons

Today we have 1. If we think this PR is just for VERY RARE use cases where the client and the relay are probably coming from the same developer, then this makes sense.

If we want to see more usage of this, then we have to go for 2 or 3. Since we like simplicity, 4 is not desired on Nostr.

[staab]

Feature support signaling is a hassle, but that's sort of the only downside. It would be nice to make any additions signal-able by putting them in a new NIP, so relays can signal support if they want to (even though many don't).

[fiatjaf]

The most important downside of feature signaling is that it makes changing things a much easier process than currently.

[bezysoftware]

The most important downside of feature signaling is that it makes changing things a much easier process than currently.

But currently feature signaling is done by adding new NIPs, so.. I don't follow. Do you mean we should have a different way to signal feature support?

[bezysoftware]

FYI I have this implemented in netstr relay and deployed to a dev instance: wss://relay-dev.netstr.io
Feel free to play with it (e.g. according to meme sample: ["REQ", "test", { "&t": ["meme", "cat"], "#t": ["white", "black"] }]

[dskvr]

What's worse than support signaling? A NIP that is merged too quickly and changes n times.

IMO Just leave this open and let relays signal support for it, like is already happening. If it works and is found helpful, the benefits of a merge will be implicit, the need for discussion will be reduced and the effort of merging will be minimized.

I see no reason to rush this NIP (or really, most NIPs).

[bezysoftware]

I'm personally happy with this NIP and deployed it to production: https://relay.netstr.io/

[lez]

I want to add a use-case for the possible use of this NIP.
If someone was to create a deli.cio.us or pinboard.in like public bookmarking tool over nostr, this NIP would be practical to filter other people's bookmarks based on tags, like #nostr #messaging #android

Is there any objection left here? When is it going to be merged?

[mikedilger]

Instead of the special rules at the end about not mixing ANDs and ORs, this could have been specified in disjunctive normal form as an array of arrays, where the inner values are ANDed and then the array of those terms is ORed. Just a thought, not a push for a proposal or anything.

[hzrd149]

Opened PR to nostr-tools for JavaScript nbd-wtf/nostr-tools#518

[dskvr]

NIP-ND -> NIP-91

[hzrd149]

Built an example in applesauce for https://hzrd149.github.io/applesauce/examples/#hashtags/explore

[vitorpamplona]

I just realized this is great for decrypted NIP-17 group chats where we need to filter by the presence of many p-tags in local dbs.

@jb55 since you are putting them on nostrdb, have you thought about indexing in this way?

[vitorpamplona]

Added to Amethyst/Quartz

[Semisol]

Next Nostr.land relay software release will support this

[dskvr]

This NIP now has wider support than the majority of approved NIPs did before they were merged, and more support than a handful of the "approved" NIPs do years after they were merged.

[vitorpamplona]

@dskvr have you thought about a similar filter but limited by only the mentioned tags?

For instance, I want all the kind 1s that cited Myself and Amethyst's key in the same event, but no other p tags.

[dskvr]

@vitorpamplona I will not be making any functional amendments to this NIP.

[fiatjaf]

Sorry, this shouldn't have been merged. Rewriting history now.

[dskvr]

HAHAHAHAHAHAHAHAHAHAHAHA

@staab @vitorpamplona @eskema could you reopen it please?

[staab]

I don't have the button, you'll have to open a new one.

[dskvr]

@fiatjaf for all the "centralization slippery slope" talk you are definitely not leading by example by rewriting history because you don't like a widely-adopted NIP 🥇

[staab]

Is it widely adopted though? I see one relay signaling support on nostr.watch

[hzrd149]

@fiatjaf @staab if this wasn't supposed to be merged then it at least should have been left open so the conversation could continue. I don't see why it was necessary to merge then revert / force push

[staab]

@hzrd149 yes, I agree. I merged because this looks good and technically meets the requirements in the readme, but fiatjaf is right that it isn't really "widely adopted" (at least I don't see any evidence that it is). I don't want to argue about it, I'm not interested in this NIP so I'm just not going to participate any more, but I can't reopen because github won't let me.

[dskvr]

@staab @fiatjaf

Is it widely adopted though?

NIP-32 "OK!"

NIP-91 "not widely adopted"

Final Outcome

• merge
• rewrite history
• "not interested in this NIP, not participating"
• PR buried

[staab]

That's what draft PRs are for you melon. There were several implementations by the time it was merged.

[alexgleason]

It's a good idea that should have been part of NIP-01 to begin with. But the problem with it is that it changes the NostrFilter interface, one of the 2 only core interfaces to the Nostr protocol. Meaning that every Nostr library in existence would have to be updated. At this point I think it would be exciting to see a big change like this, though. Give Nostr devs something to do lol.

[dskvr]

@alexgleason The way applesauce did it was simple and elegant. Clients can support AND without relays supporting AND, the library just does the intersections locally when relay doesn't have support, not different from how client-developer has to do right now. Instead of AND being implemented by hand by each developer, it can be signaled by a protocol level operand, offloaded to the library's post-filters; something that already exists in every single library. If relays implement NIP-01 correctly, then they just ignore the filter. It's voluntary. It breaks nothing. Find me a case that breaks anything, for any properly and defensively implemented client, relay or library; I will happily eat my words. As libraries and relays support it, clients adopt it, and everbody wins. Every. Single. Stakeholder: Relay Operators, Client Developers and Users.

---

There's an open PR to nostr-tools, but it will never be merged. There are also 2 open PRs to strfry, but the one that was open before all this unfolded was dismissed for inclusion because someone told doug not to because "there are no use cases" and "any use case that uses #t is dumb" (paraphrased) where the former is objectively false, and the latter is absurd and is not a real argument. The only arguments that might have merit is that it is a "centralization risk" and a potential "slippery slope," but this has never been explained and is purely theoretical. Saying something over and over doesn't immediately make it true, even if you have been ordained a high-priest. If AND is a slippery slope, why isn't OR a slippery slope?

I didn't even expect this to be merged, basically ever, I said that much a year ago higher up in the thread. At least until the acceptance was wide enough that it was obvious.

But how things have unfolded speak louder than words:

1. Actively uses clout to deter developers from implementing things he doesn't like; not logical arguments, blatant manipulation of facts and/or ad-hominem (IE: social attack vector)
2. Rewrites history and force pushes on a "decentralized" protocol.
3. (1 day) After actively suppressing the NIP from being considered for implementation on the most popular relay, says this NIP "has no support" to justify actions.
4. Force push makes it impossible to reopen PR, forcing a conversational reset and resetting all numbers and age of the NIP.

^ 1-4 occurred over a time-period of 48 hours.

This is the kind of behavior one expects from despots, not the maintainer of a self-proclaimed "decentralized" and "self-sovereign" protocol.

If nostr-protocol/nips had a canary, this PR is it.