exposing a region's configuration/lock for custom functionality #125
Description
Migrated issue, originally created by jvanasco (jvanasco)
with some advanced Redis usage, I need to handle caching certain keys in ways that are incompatible with dogpile.cache as-is, and must use the dogpile.core lock directly. at the same time, I still want to leverage my dogpile region and it's configuration.
by "incompatible", i mean various mechanisms that can't be fixed with a custom backend or Proxy. In one example, a key corresponds to a redis hash with messages for a user, and contains at least 3 values (count unread, count read, cached_ids):
messages|{user_id} count_read = 10
messages|{user_id} count_unread = 5
messages|{user_id} cached_ids = (1,2,3,4,5,6,7,8,9,10)
messages|{user_id} msg-1 = {foo: bar}
messages|{user_id} msg-2 = {foo: bar}
messages|{user_id} msg-3 = {foo: bar}
getting/updating the message counts requires writing/validating two keys at once (count_read, count_unread). various functions need to happen within a redis pipeline as well.
to handle this sort of operation, I am essentially leveraging the code below - which is based on the getcreate logic
would it make sense to integrate something like this into the CacheRegion itself?
#!python
import logging
log = logging.getLogger(__name__)
from dogpile import Lock, NeedRegenerationException
from dogpile.cache.api import NO_VALUE
def raw_getcreate(region, key, f_get, f_gen, f_async=None):
# key is needed for the lock
def _get_value():
if __debug__:
log.debug("raw_getcreate._get_value")
value = f_get()
if value in (None, NO_VALUE):
raise NeedRegenerationException()
return value, -1
def _gen_value():
if __debug__:
log.debug("raw_getcreate._gen_value")
try:
if __debug__:
log.debug("raw_getcreate._gen_value | first, try fetching")
return _get_value()
except NeedRegenerationException as e:
pass
if __debug__:
log.debug("raw_getcreate._gen_value | create")
created_value = f_gen()
return created_value, -1
with Lock(
region._mutex(key),
_gen_value,
_get_value,
None, # expiration_time, never expire
f_async, # async creator, unused
) as value:
return value
reg = CACHE_REGIONS['foo']
key = "testkey-2"
def f_get():
value = reg.backend_actual.get(key)
return value
def f_gen():
created_value = "NEWDATA"
uploaded = reg.backend_actual.set(key, created_value)
return created_value
print raw_getcreate(reg, key, f_get, f_gen)