Array view returning wrong length

[kpiwonski]

Hi,

Consider below example:

    let table = Table::new(
        "users".into(),
        vec![
            FieldArray::from_arr("id", arr_i32![1, 2, 3]),
            FieldArray::from_arr("name", arr_str32!["alice", "bob", "charlie"]),
        ]
        .into(),
    );
    // let v = table.r(&[1, 2]);
    let v = table.r(1..=2);
    let vv: &[i32] = &v.cols[0].array.inner::<IntegerArray<i32>>();
    dbg!(vv.len());

In this case debug will return 3. Expected value is 2. However, while using let v = table.r(&[1, 2]) it would return length of 2 as expected. I think the expected outcome is the result should be consistent and return same value as they give effectively same indexes to filter.

[pbower]

Hi @kpiwonski ,

Thanks for raising this.

This is expected behaviour, though I appreciate it's not immediately obvious. The two selection styles work differently under the hood, due to the contiguous slice vs. (often) non-contiguous selection styles.

Please see the explanation below:

Overview

1. table.r(1..=2) - range selection, which is known to be contiguous at the type level, so it creates a zero-copy view.

• When you access .array on the resulting ArrayV, you're reaching through the view to the full parent array (all 3 elements). The windowing is tracked separately via offset and length:

  pub struct ArrayV {
      pub array: Array, // the full parent array, contains Arc<inner>
      pub offset: usize,
      len: usize,
      null_count: OnceLock<usize>,
  }

This structure allows for zero-copy access where ArrayV manages the offsets and indexing, as though it were not still the original array, enabling efficient windowed access.

• The view abstraction does not extend down to the inner variant types (e.g., IntegerArray) to keep compile times fast and help maintain simplicity.

2. table.r(&[1, 2]) - index array selection, which cannot be assumed contiguous at the type level, so it materialises a new table by gathering the selected rows. The backing array genuinely has 2 elements because it was freshly allocated. (A runtime contiguity check could optimise cases like &[1, 2] into the view path, but the benefit is marginal for typical workloads.)

Solution

To get the windowed length, use v.cols[0].len() rather than going through .array.inner::<T>(). Or if what you want is to work with the underlying typed IntegerArray, this is the route:

  let v = table.r(1..=2);
  let arr = v.cols[0].to_array();
  let data: &IntegerArray<i32> = arr.inner::<IntegerArray<i32>>();
  assert_eq!(data.len(), 2);

Note that to_array() copies the windowed data, as the inner array types are
owned.

Alternatively, if you just need the values as a typed slice without the full array wrapper, you can use to_typed_vec:

  let v = table.r(1..=2);
  let data: Vec64<i32> = v.cols[0].to_typed_vec::<i32>().unwrap();
  assert_eq!(data.len(), 2);
  assert_eq!(data.as_slice(), &[2, 3]);
// And if you need an IntegerArray from that:
  let int_arr = IntegerArray::<i32>::from(data);

I'm open to renaming ArrayV's .array (field member) to .outer_array to make it clearer that it's the full backing array rather than the windowed slice, if you'd find that helpful ?

Regards,
Peter