Feature: Add NIP-91

[dskvr]

NIP-91 AND tag filters

Intent

• Implement NIP-91-style AND semantics for tag filters using &<tag> keys while preserving existing OR behavior for #<tag>.
• Keep the DB query engine, monitor engine and Perl reference filter in sync on tag semantics without regressing performance.

Changes

• src/filters.h:
  • Extend NostrFilter with tagsAnd to represent ANDed tag values and track unique tag keys across # and &.
  • Parse &<tag> keys, hex-decode &e/&p, and normalise values via FilterSetBytes, de-duplicating overlaps so values present in & are removed from the corresponding #.
  • Adjust indexOnlyScans so presence of any AND tag forces full-event scans and keeps the 3-tag-key limit based on the union of # and & keys.
• src/DBQuery.h:
  • Allow tag-based scans to be driven by either tags or tagsAnd, choosing the most selective tag key.
  • When scanning by an AND tag, use only one value from the AND set for the LMDB cursor and disable index-only mode (full events are re-checked against the remaining AND values).
• src/ActiveMonitors.h:
  • Include tagsAnd alongside tags in the tag monitor index so subscription monitors receive events that satisfy AND tag filters.
• test/dumbFilter.pl:
  • Replace hard-coded #e/#p/#t logic with generic OR (#<tag>) and AND (&<tag>) handling to match the NostrFilter implementation, including ignoring values that appear in both AND and OR for the same tag key.
• test/filterFuzzTest.pl:
  • Extend fuzzed filters to include &t, &e and &p so scan and monitor fuzz tests cover NIP-91 AND-tag semantics.

Testing

From the project root:

• Build the binary: make.
• Populate a test database as described in test/README.md (for example, import the wellordered 500k dataset).
• Run the existing fuzz tests, which exercise the new AND-tag logic:
  • perl test/filterFuzzTest.pl scan-limit
  • perl test/filterFuzzTest.pl scan
  • perl test/filterFuzzTest.pl monitor

[hoytech]

I asked @fiatjaf about his thoughts on this, and he's not convinced it's a generally useful feature. I'm somewhat inclined to agree, since we don't want to slide down the slippery slope of re-implementing a crappy version of SQL!

Overall, my first impression of the diff is that the code looks reasonable, so I'm not necessarily opposed to this. However, I am a bit nervous about making such a significant change to the core filtering logic.

[dskvr]

asked @fiatjaf about his thoughts on this, and he's not convinced it's a generally useful feature.

Don't let this bother you, enough good developers support the general idea, and it's been implemented.

See here: nostr-protocol/nips#1365

we don't want to slide down the slippery slope of re-implementing a crappy version of SQL!

I don't want that either. There is the belief that adding this is a slippery slope into ranges, math operands, IS NULL, IS NOT NULL, etc). Which is why the NIP was intentionally written in a limiting way.

Right now if I want all memes that are of a black cat I have to stream the entire haystack from a relay to make the intersection and find the needle; which people are doing right now for real cases like NIP-17.

Not a single stakeholder wins from this:

1. Relay operators lose clock-time and bandwidth
2. Developers have significantly higher maintenance requirements for this case and an even harder optimization problem
3. users watch their mobile data disappear in a single interaction.

However, I am a bit nervous about making such a significant change to the core filtering logic.

This is why I have left it as a draft; I am honestly not certain my implementation is valid or good.