Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
631 commits
Select commit Hold shift + click to select a range
10c5f7c
added logging and fixed tests
jpesenhofer Apr 26, 2023
bb502df
retutn correct offset
jpesenhofer Apr 27, 2023
8e96922
removed additional plus
jpesenhofer Apr 27, 2023
496901b
removed logs
jpesenhofer Apr 27, 2023
32261ee
put flush in own function
jpesenhofer Apr 27, 2023
6d9211c
flush on revoked partition
jpesenhofer Apr 27, 2023
67137b3
removed offsets from cache
jpesenhofer Apr 27, 2023
9dd45b1
removed some functions
jpesenhofer Apr 27, 2023
9e98cb1
added value to check exception on set
jpesenhofer Apr 27, 2023
f43b84c
turn off batcher
jpesenhofer Apr 27, 2023
f5af306
added batcher again with different mutation approach
jpesenhofer Apr 27, 2023
e747b54
revival of the mutation buffer
jpesenhofer Apr 28, 2023
b2b0f8f
fixed some stuff in mutation buffer
jpesenhofer Apr 28, 2023
782c2da
remove items method
jpesenhofer Apr 28, 2023
64f60d3
added mutatation rows and values in init
jpesenhofer Apr 28, 2023
18ceeb4
fixed list cast
jpesenhofer Apr 28, 2023
2987e47
removed force
jpesenhofer Apr 28, 2023
675afe5
fixed tests and moved mutation buffer to cache
jpesenhofer May 23, 2023
0b7a24c
fixed flush mutations
jpesenhofer May 24, 2023
a91b65c
flush after apply changelog batch
jpesenhofer May 24, 2023
63013af
added warning with traceback
jpesenhofer May 24, 2023
63e8308
fixed mutation rows
jpesenhofer May 24, 2023
f3d9ee1
changed min flush mutations
jpesenhofer May 25, 2023
bb66e30
refactored log
jpesenhofer May 25, 2023
c1736e4
try faster iteritems
jpesenhofer May 25, 2023
27977cb
don't write offsets in the cache
jpesenhofer May 25, 2023
e21b158
removed set after get
jpesenhofer May 25, 2023
3ef552d
fixed iterkeys
jpesenhofer May 25, 2023
cdb7536
fixed iteritems again and added logs for revoking and assigning
jpesenhofer May 26, 2023
50974ba
mitigate race conditions while logging
jpesenhofer May 26, 2023
e3dba58
adjusted logging and faster iteritems if value cache is enabled
jpesenhofer May 26, 2023
0ebe9c5
always flush.
jpesenhofer May 26, 2023
9744db4
move reset of total mutation count up
jpesenhofer May 26, 2023
2d7984b
fixed logging for iteritems
jpesenhofer May 26, 2023
f0ddb7d
in get range we want to be aware of the fact, that deletes are done on
jpesenhofer Jun 7, 2023
53b2bae
remove flush on cache miss
jpesenhofer Jun 7, 2023
812bada
hardcode partition cache
jpesenhofer Jun 13, 2023
85c08fe
added todo
jpesenhofer Jun 13, 2023
a6e4720
removed logs
jpesenhofer Jun 13, 2023
929c920
fixed get partition from cache
jpesenhofer Jun 13, 2023
9054a91
fixed wrong use of partition cache
jpesenhofer Jun 13, 2023
600cd53
don't use get many
jpesenhofer Jun 13, 2023
e656e8f
faster get from cache and smaller bitable get function
jpesenhofer Jun 13, 2023
71db278
adjust parameters and removed value cache completely
jpesenhofer Jun 14, 2023
43750a5
refactored settings
jpesenhofer Jun 14, 2023
987b782
added options to logging
jpesenhofer Jun 14, 2023
61d7e96
added loggingh
jpesenhofer Jun 14, 2023
7a97906
removed log
jpesenhofer Jun 14, 2023
33fabab
fixed tests
jpesenhofer Jun 14, 2023
4273b96
fixed get with wrong partitions
jpesenhofer Jun 15, 2023
235df0a
removed contains any
jpesenhofer Jun 15, 2023
7ffad62
fixed wrong return value if no partition was found and cache was
jpesenhofer Jun 15, 2023
fe901fb
catch exception on mutation flush
jpesenhofer Jun 16, 2023
b14ade7
changed default partition name
jpesenhofer Jun 16, 2023
223bdbf
fixed contains call
jpesenhofer Jun 16, 2023
266e245
adjusted tests ✅
jpesenhofer Jun 16, 2023
108497e
flush on all submitted mutations
jpesenhofer Jun 22, 2023
e3a1175
update mutation size to 10k
jpesenhofer Jun 22, 2023
1050b15
only set if value is not None in value set
jpesenhofer Jun 22, 2023
2ad775c
changed to 1000 mutations
jpesenhofer Jun 22, 2023
6323415
top
jpesenhofer Jun 22, 2023
8fb9ce5
push max mutations to the max
jpesenhofer Jun 22, 2023
8cb56fa
submit directly
jpesenhofer Jun 22, 2023
7059218
fixed contains of value cache
jpesenhofer Jun 23, 2023
d007287
try different approach on contains
jpesenhofer Jun 23, 2023
95640b4
remove cache access entirely
jpesenhofer Jun 23, 2023
e12ec73
fix something in asssign partitions where the standby partition was also
jpesenhofer Jun 23, 2023
1b44c27
added logging in get requests
jpesenhofer Jun 23, 2023
25251fb
fix offset
jpesenhofer Jun 23, 2023
25055f8
fixed wrong offset value
jpesenhofer Jun 23, 2023
cfeb4d1
Squashed commit of the following:
jpesenhofer Jul 5, 2023
ebcae09
removed partition assigned and rebalance logic
jpesenhofer Jul 5, 2023
7cf68ee
removed partition prefix
jpesenhofer Jul 5, 2023
d34ca0e
remove partitioning
jpesenhofer Jul 7, 2023
e2b5560
adjusted batch read for changelogs
jpesenhofer Jul 7, 2023
d258ac7
used delete in changelog recovery
jpesenhofer Jul 7, 2023
140bc6a
fixed wrong offset key for comparison
jpesenhofer Jul 7, 2023
c16e892
dont cache all mutations
jpesenhofer Jul 7, 2023
10f318e
remove cache and rebalance functions
jpesenhofer Jul 7, 2023
b953e28
added cache
jpesenhofer Jul 7, 2023
703309f
add value to cache for read item
jpesenhofer Jul 7, 2023
1c08130
added mutation buffer again
jpesenhofer Jul 10, 2023
6829b53
take mutation with iteritems
jpesenhofer Jul 10, 2023
0e1763a
continue on beginning if offset key
jpesenhofer Jul 10, 2023
8e3e4be
HARDCODED MUTATION BUFFER SETTINGS
jpesenhofer Jul 10, 2023
dc676f5
added log to check if mutations are working
jpesenhofer Jul 10, 2023
228f421
refactored if
jpesenhofer Jul 10, 2023
8e5016a
more access to mutation buffer
jpesenhofer Jul 10, 2023
b8e69b5
formatted
jpesenhofer Jul 10, 2023
8728cd1
set to 0 manually and clear all mutations
jpesenhofer Jul 10, 2023
b38c2fd
hardcoded cache
jpesenhofer Jul 10, 2023
97c73cd
copy mutation buffer
jpesenhofer Jul 11, 2023
d297282
add everything to cache because we can
jpesenhofer Jul 11, 2023
10a52b2
Squashed commit of the following:
jpesenhofer Jul 12, 2023
eeaa371
fixed rowset
jpesenhofer Jul 12, 2023
776eb66
fixed wrong iteration number
jpesenhofer Jul 12, 2023
bbda908
fixed apply changelog in bigtable
jpesenhofer Jul 18, 2023
afff96d
🩹 fixed bigtable delete testcase
jpesenhofer Sep 21, 2023
39c7094
removed pdb
jpesenhofer Sep 21, 2023
4a06e14
🩹 fixed bigtable set test and formating
jpesenhofer Sep 21, 2023
5a0edcf
✨ formating of bigtable store
jpesenhofer Sep 21, 2023
30d51ac
🩹 fixed testcase for get current partitions
jpesenhofer Sep 21, 2023
0667b90
🩹 fixed bug with wrong removal of partition
jpesenhofer Sep 21, 2023
7607eac
🩹 fix test get partitions for current key
jpesenhofer Sep 21, 2023
224b3a2
🩹 fixed get testcases
jpesenhofer Sep 21, 2023
8b7af4c
🚮 delete partition stuff from get call
jpesenhofer Sep 21, 2023
3ebb53d
🩹 fixed testcases for set and del
jpesenhofer Sep 21, 2023
ea1bb1d
🩹 fixed iteritems call
jpesenhofer Sep 21, 2023
9dd31f9
🩹 fixed testcase for inserting new offset
jpesenhofer Sep 21, 2023
805b86a
✨ refactored set_persited_offset function
jpesenhofer Sep 21, 2023
f7b43e2
🩹 fixed apply changelog batch in BT
jpesenhofer Sep 22, 2023
37287f1
reduced number for max num mutations
jpesenhofer Sep 22, 2023
b8270ae
check offset key in read rows
jpesenhofer Sep 25, 2023
d23e5d6
decode offset key before iteration
jpesenhofer Sep 26, 2023
88de63b
reduce mutation buffer and increase cache
jpesenhofer Sep 26, 2023
9a11ef4
faster changelog read
jpesenhofer Sep 26, 2023
613ff26
switch off mutattion buffer
jpesenhofer Sep 26, 2023
37f8e8c
always use directrow
jpesenhofer Sep 26, 2023
52c3082
removed no mutate, added timer for flush
jpesenhofer Sep 26, 2023
f9bc4ce
fixed unit tests and removed unused imports
jpesenhofer Sep 26, 2023
3ddbd9b
use no caches at all
jpesenhofer Sep 26, 2023
761bae1
Merge remote-tracking branch 'origin/sx-global-global-tables' into re…
jpesenhofer Sep 27, 2023
b5e342c
faster iteritems
jpesenhofer Sep 27, 2023
5eee149
faster iteritems
jpesenhofer Sep 27, 2023
abd1612
add exception for existing table exception
jpesenhofer Sep 27, 2023
1bdb7b0
log which partitions were used for iteritems
jpesenhofer Sep 28, 2023
4141eca
removed comma
jpesenhofer Sep 28, 2023
f8f972f
fill value cache on startup
jpesenhofer Sep 28, 2023
2015381
added set
jpesenhofer Sep 28, 2023
a726025
added logging and use proper partitions
jpesenhofer Sep 28, 2023
dabd88c
added keycache
jpesenhofer Sep 29, 2023
2f0da12
removed key cache from contains
jpesenhofer Sep 29, 2023
458ed9e
reduce contains
jpesenhofer Sep 29, 2023
b51806b
added revoke partitons and utility function
jpesenhofer Sep 29, 2023
aa7650a
addde logging
jpesenhofer Sep 29, 2023
7b86625
remove revoke partition
jpesenhofer Sep 29, 2023
cac2ddc
add first version of mutation buffer again
jpesenhofer Sep 29, 2023
e6e5030
moved setup of mutation buffer
jpesenhofer Sep 29, 2023
bf9da1a
fixed intendation
jpesenhofer Sep 29, 2023
e43c692
fixed stuff
jpesenhofer Sep 29, 2023
6fc8b0e
fixed bug
jpesenhofer Sep 29, 2023
b9907eb
formatting
jpesenhofer Sep 29, 2023
616fae5
logging for flush and get only from table directly
jpesenhofer Sep 29, 2023
dca46b4
updated startup cache also in apply changelogtopics
jpesenhofer Sep 29, 2023
924137f
removed mutation buffer ✨
jpesenhofer Sep 29, 2023
c5c6293
made fields configurable
jpesenhofer Oct 9, 2023
e0a1fc4
fixed some tests for configurations
jpesenhofer Oct 9, 2023
b3dd33d
changed flush log
jpesenhofer Oct 9, 2023
ec7550c
updated key and startup cache correct on startup
jpesenhofer Oct 10, 2023
ed32cc9
check caches in _contains
jpesenhofer Oct 10, 2023
383594a
added mutation cache
jpesenhofer Oct 10, 2023
0f370bb
Revert "added mutation cache"
jpesenhofer Oct 10, 2023
6affe29
invalidate startup cache on set
jpesenhofer Oct 11, 2023
412004e
removed checks in _contains
jpesenhofer Oct 11, 2023
1189e9b
implemented new caching logic
jpesenhofer Oct 11, 2023
35e58e5
deleting the startup cache is only done by the invalidation timer
jpesenhofer Oct 11, 2023
2b772b9
added logging for invalidating cache
jpesenhofer Oct 11, 2023
a9a7d89
add logging
jpesenhofer Oct 12, 2023
f3b7883
added more logging
jpesenhofer Oct 12, 2023
a6290e5
fixed syntax error
jpesenhofer Oct 12, 2023
8f1c84a
fixed logging
jpesenhofer Oct 12, 2023
50a52ed
if table is globa or uses partitioner, try all partitions
jpesenhofer Oct 12, 2023
0c956fa
try differetn approach on _get_current_partitions
jpesenhofer Oct 12, 2023
43c4cba
fixed wrong apply changelog batch
jpesenhofer Oct 12, 2023
10eea20
fixed apply changelog topics again
jpesenhofer Oct 12, 2023
27f82de
fixed error in set
jpesenhofer Oct 12, 2023
549a3b6
use msg partition for adding data to table
jpesenhofer Oct 12, 2023
67cfe9e
fix set for no key translation
jpesenhofer Oct 12, 2023
af84131
fix partitioning
jpesenhofer Oct 12, 2023
0f5f324
removed no row found log
jpesenhofer Oct 12, 2023
a63c754
changed set get delete and contains method
jpesenhofer Oct 17, 2023
feb2937
fixed error
jpesenhofer Oct 17, 2023
33c3045
fixed wrong offset key get
jpesenhofer Oct 17, 2023
b444546
get offset from bt always
jpesenhofer Oct 17, 2023
692fd1b
added logging msg for missed key
jpesenhofer Oct 17, 2023
ccc5226
removed unused function
jpesenhofer Oct 18, 2023
d59cf70
moved log
jpesenhofer Oct 18, 2023
4ebaafe
call get in contains
jpesenhofer Oct 18, 2023
1818c44
try different logging REVERT ME
jpesenhofer Oct 18, 2023
b8ef8a2
Revert "try different logging REVERT ME"
jpesenhofer Oct 18, 2023
5405ec7
Revert "Revert "try different logging REVERT ME""
jpesenhofer Oct 18, 2023
6590c68
more logging
jpesenhofer Oct 18, 2023
cf25c2a
fixed syntax
jpesenhofer Oct 18, 2023
898a061
more logging
jpesenhofer Oct 18, 2023
b84144d
assert no topic in set
jpesenhofer Oct 18, 2023
9d3acd9
add check for topc in event when searching for current event
jpesenhofer Oct 18, 2023
0f3022c
removed assert
jpesenhofer Oct 18, 2023
2ce0a32
removed flush log
jpesenhofer Oct 18, 2023
14836fc
removed log and temporary debugging
jpesenhofer Oct 18, 2023
4a76dec
removed keycache and refactored value cache
jpesenhofer Oct 19, 2023
a181095
formatted
jpesenhofer Oct 19, 2023
0a2b58e
flush mutation buffer on stop
jpesenhofer Oct 19, 2023
9e234bc
always start new thread
jpesenhofer Oct 19, 2023
696fb9f
don't start in invalidate startup cache
jpesenhofer Oct 19, 2023
b8cf623
delete invalidation timer
jpesenhofer Oct 23, 2023
c0b6392
set invalidation timer to None
jpesenhofer Oct 23, 2023
7c01ebf
fill caches after reading changelog
jpesenhofer Oct 23, 2023
29dc940
Merge pull request #4 from smaxtec/sx-global-global-tables
Mopsgeschwindigkeit Nov 2, 2023
d45808d
add startupcache partitions for faster iteritems
jpesenhofer Nov 2, 2023
eacf2cf
only yield from bigtable if partitions are left over
jpesenhofer Nov 2, 2023
8034689
fixed typing
jpesenhofer Nov 2, 2023
73510af
fix syntax error
jpesenhofer Nov 2, 2023
d085439
removed type declaration
jpesenhofer Nov 2, 2023
75c0ee7
added ttl setting
jpesenhofer Nov 6, 2023
de30378
removed all unit tests
jpesenhofer Nov 8, 2023
94cbbd1
removed all tests
jpesenhofer Nov 8, 2023
7c72248
further progress with tests
jpesenhofer Nov 10, 2023
9b241bd
fixed bug in delete where delete was called with a wrong key
jpesenhofer Nov 10, 2023
d7a6b9d
fixed unit tests for bigtable
jpesenhofer Nov 10, 2023
6cdd356
added additional tests and fixed stuff in the implementation
jpesenhofer Nov 10, 2023
54d3868
faster get requests with abort for startup cache values
jpesenhofer Nov 13, 2023
c4de292
added tests to increase coverage
jpesenhofer Nov 13, 2023
782faf9
added addtional unit tests
jpesenhofer Nov 13, 2023
cf863f8
Merge branch 'sxmaster' into bigtable-store
jpesenhofer Nov 13, 2023
b3adfa4
added mutation batcher cache
jpesenhofer Nov 14, 2023
04eada7
added unit tests for set mutation and for iterating global tables
jpesenhofer Nov 14, 2023
bdf92e4
added more unit tests and also increased coverage
jpesenhofer Nov 14, 2023
6b5d709
removed unused import
jpesenhofer Nov 14, 2023
608a6a5
moved log for assigning partitions
jpesenhofer Nov 15, 2023
3e4ab65
removed unnecesery abort on _get
jpesenhofer Nov 15, 2023
8a6cec1
fix naming of extract row data function
jpesenhofer Nov 15, 2023
6079a65
Update install
jpesenhofer Nov 20, 2023
cb07b92
Merge pull request #2 from smaxtec/bigtable-store
jpesenhofer Nov 21, 2023
c30e7f5
fixed return value and added testcase
jpesenhofer Nov 27, 2023
7b760a4
Merge pull request #8 from smaxtec/fix-wrong-return-in-bigtable-get
jpesenhofer Nov 27, 2023
5291ac0
fixed bug with mutation batcher in get requests if a delete mutation …
jpesenhofer Nov 30, 2023
06b3997
added partition revoking and log
jpesenhofer Dec 1, 2023
e16dcc7
Merge pull request #9 from smaxtec/add_revoke_partitions
jpesenhofer Dec 1, 2023
66df010
revoke all tps also the ones that are not active
jpesenhofer Dec 12, 2023
2096f40
Merge pull request #10 from smaxtec/fix_revoke_partitions_for_bigtable
jpesenhofer Dec 12, 2023
3266ae7
moved logging and added todo
jpesenhofer Dec 15, 2023
afbca06
Merge pull request #11 from smaxtec/debounce-logging-in-bigtable
jpesenhofer Dec 15, 2023
a6e5e0f
added option and added tests
jpesenhofer Dec 19, 2023
5db83dc
Merge pull request #12 from smaxtec/add-option-for-endless-startup-cache
jpesenhofer Dec 19, 2023
4f50676
updated unit tests and bigtable store
jpesenhofer Dec 19, 2023
32dc779
added some logs
jpesenhofer Dec 19, 2023
d8b2cce
added gc collect
jpesenhofer Dec 19, 2023
4b7384b
introduced locking on rebalance and fixed unit tests
jpesenhofer Dec 19, 2023
199949c
fill startup cache after recovery
jpesenhofer Dec 19, 2023
2e7730e
made some adjustments to the startup cache and fixed tests accordingly
jpesenhofer Dec 19, 2023
7b68855
adjusted some more stuff
jpesenhofer Dec 19, 2023
ecb1269
create no existing partitions on assign in value cache
jpesenhofer Dec 20, 2023
711b615
also create empty cache if partition is not in startup cache already
jpesenhofer Dec 20, 2023
cb095f5
fixed bug
jpesenhofer Dec 20, 2023
2d338e9
removed log
jpesenhofer Dec 21, 2023
8092517
Merge pull request #14 from smaxtec/faster-bigtable-startup-revokes
jpesenhofer Jan 17, 2024
2661270
Revert "Merge pull request #14 from smaxtec/faster-bigtable-startup-r…
jpesenhofer Jan 17, 2024
e4d50d6
Merge pull request #15 from smaxtec/revert-bigtable-merge
jpesenhofer Jan 18, 2024
7206241
Merge remote-tracking branch 'upstream/master' into merge-upstream-05…
Mar 5, 2026
e760baf
Fix bigtable tests and aiokafka producer compatibility
Mar 5, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions faust/__init__.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
"""Python Stream processing."""

# :copyright: (c) 2017-2020, Robinhood Markets, Inc.
# All rights reserved.
# :license: BSD (3 Clause), see LICENSE for more details.
Expand Down
1 change: 1 addition & 0 deletions faust/stores/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
memory="faust.stores.memory:Store",
rocksdb="faust.stores.rocksdb:Store",
aerospike="faust.stores.aerospike:AeroSpikeStore",
bigtable="faust.stores.bigtable:BigTableStore",
)
STORES.include_setuptools_namespace("faust.stores")
by_name = STORES.by_name
Expand Down
10 changes: 5 additions & 5 deletions faust/stores/aerospike.py
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ def _get(self, key: bytes) -> Optional[bytes]:
key = (self.namespace, self.table_name, key)
fun = self.client.get
try:
(key, meta, bins) = self.aerospike_fun_call_with_retry(fun=fun, key=key)
key, meta, bins = self.aerospike_fun_call_with_retry(fun=fun, key=key)
if bins:
return bins[self.BIN_KEY]
return None
Expand Down Expand Up @@ -173,7 +173,7 @@ def _itervalues(self) -> Iterator[bytes]:
fun=fun, namespace=self.namespace, set=self.table_name
)
for result in scan.results():
(key, meta, bins) = result
key, meta, bins = result
if bins:
yield bins[self.BIN_KEY]
else:
Expand All @@ -193,8 +193,8 @@ def _iteritems(self) -> Iterator[Tuple[bytes, bytes]]:
fun=fun, namespace=self.namespace, set=self.table_name
)
for result in scan.results():
(key_data, meta, bins) = result
(ns, set, policy, key) = key_data
key_data, meta, bins = result
ns, set, policy, key = key_data

if bins:
bins = bins[self.BIN_KEY]
Expand All @@ -214,7 +214,7 @@ def _contains(self, key: bytes) -> bool:
try:
if self.app.conf.store_check_exists:
key = (self.namespace, self.table_name, key)
(key, meta) = self.aerospike_fun_call_with_retry(
key, meta = self.aerospike_fun_call_with_retry(
fun=self.client.exists, key=key
)
if meta:
Expand Down
Loading
Loading