unbound: util/storage/lruhash.h File Reference

unbound 0.1

Data Structures | Macros | Typedefs | Functions
lruhash.h File Reference

This file contains a hashtable with LRU keeping of entries. More...

`#include "util/locks.h"`

## Data Structures
struct | lruhash
| Hash table that keeps LRU list of entries. More...
struct | lruhash_bin
| A single bin with a linked list of entries in it. More...
struct | lruhash_entry
| An entry into the hash table. More...

## Macros
#define | HASH_DEFAULT_STARTARRAY 1024 /* entries in array */
| default start size for hash arrays
#define | HASH_DEFAULT_MAXMEM 4*1024*1024 /* bytes */
| default max memory for hash arrays

## Typedefs
typedef uint32_t | hashvalue_type
| the type of a hash value
typedef size_t(* | lruhash_sizefunc_type) (void *, void *)
| Type of function that calculates the size of an entry.
typedef int(* | lruhash_compfunc_type) (void *, void *)
| type of function that compares two keys.
typedef void(* | lruhash_delkeyfunc_type) (void *, void *)
| old keys are deleted.
typedef void(* | lruhash_deldatafunc_type) (void *, void *)
| old data is deleted.
typedef void(* | lruhash_markdelfunc_type) (void *)
| mark a key as pending to be deleted (and not to be used by anyone).

## Functions
struct lruhash * | lruhash_create (size_t start_size, size_t maxmem, lruhash_sizefunc_type sizefunc, lruhash_compfunc_type compfunc, lruhash_delkeyfunc_type delkeyfunc, lruhash_deldatafunc_type deldatafunc, void *arg)
| Create new hash table.
void | lruhash_delete (struct lruhash *table)
| Delete hash table.
void | lruhash_clear (struct lruhash *table)
| Clear hash table.
void | lruhash_insert (struct lruhash *table, hashvalue_type hash, struct lruhash_entry *entry, void *data, void *cb_override)
| Insert a new element into the hashtable.
struct lruhash_entry * | lruhash_lookup (struct lruhash *table, hashvalue_type hash, void *key, int wr)
| Lookup an entry in the hashtable.
void | lru_touch (struct lruhash *table, struct lruhash_entry *entry)
| Touch entry, so it becomes the most recently used in the LRU list.
void | lruhash_setmarkdel (struct lruhash *table, lruhash_markdelfunc_type md)
| Set the markdelfunction (or NULL)
void | lruhash_update_space_used (struct lruhash *table, void *cb_override, int diff_size)
| Update the size of an element in the hashtable.
void | lru_demote (struct lruhash *table, struct lruhash_entry *entry)
| Demote entry, so it becomes the least recently used in the LRU list.
struct lruhash_entry * | lruhash_insert_or_retrieve (struct lruhash *table, hashvalue_type hash, struct lruhash_entry *entry, void *data, void *cb_arg)
| Insert a new element into the hashtable, or retrieve the corresponding element of it exits.
void | lruhash_remove (struct lruhash *table, hashvalue_type hash, void *key)
| Remove entry from hashtable.
void | bin_init (struct lruhash_bin *array, size_t size)
| init the hash bins for the table
void | bin_delete (struct lruhash *table, struct lruhash_bin *bin)
| delete the hash bin and entries inside it
struct lruhash_entry * | bin_find_entry (struct lruhash *table, struct lruhash_bin *bin, hashvalue_type hash, void *key, size_t *collisions)
| Find entry in hash bin.
void | bin_overflow_remove (struct lruhash_bin *bin, struct lruhash_entry *entry)
| Remove entry from bin overflow chain.
void | bin_split (struct lruhash *table, struct lruhash_bin *newa, int newmask)
| Split hash bin into two new ones.
void | reclaim_space (struct lruhash *table, struct lruhash_entry **list)
| Try to make space available by deleting old entries.
void | table_grow (struct lruhash *table)
| Grow the table lookup array.
void | lru_front (struct lruhash *table, struct lruhash_entry *entry)
| Put entry at front of lru.
void | lru_remove (struct lruhash *table, struct lruhash_entry *entry)
| Remove entry from lru list.
void | lruhash_status (struct lruhash *table, const char *id, int extended)
| Output debug info to the log as to state of the hash table.
size_t | lruhash_get_mem (struct lruhash *table)
| Get memory in use now by the lruhash table.
void | lruhash_traverse (struct lruhash *h, int wr, void(*func)(struct lruhash_entry *, void *), void *arg)
| Traverse a lruhash.

## Detailed Description

This file contains a hashtable with LRU keeping of entries.

The hash table keeps a maximum memory size. Old entries are removed to make space for new entries.

The locking strategy is as follows: o since (almost) every read also implies a LRU update, the hashtable lock is a spinlock, not rwlock. o the idea is to move every thread through the hash lock quickly, so that the next thread can access the lookup table. o User performs hash function.

For read: o lock hashtable. o lookup hash bin. o lock hash bin. o find entry (if failed, unlock hash, unl bin, exit). o swizzle pointers for LRU update. o unlock hashtable. o lock entry (rwlock). o unlock hash bin. o work on entry. o unlock entry.

To update an entry, gain writelock and change the entry. (the entry must keep the same hashvalue, so a data update.) (you cannot upgrade a readlock to a writelock, because the item may be deleted, it would cause race conditions. So instead, unlock and relookup it in the hashtable.)

To delete an entry: o unlock the entry if you hold the lock already. o lock hashtable. o lookup hash bin. o lock hash bin. o find entry (if failed, unlock hash, unl bin, exit). o remove entry from hashtable bin overflow chain. o unlock hashtable. o lock entry (writelock). o unlock hash bin. o unlock entry (nobody else should be waiting for this lock, since you removed it from hashtable, and you got writelock while holding the hashbinlock so you are the only one.) Note you are only allowed to obtain a lock while holding hashbinlock. o delete entry.

The above sequence is: o race free, works with read, write and delete. o but has a queue, imagine someone needing a writelock on an item. but there are still readlocks. The writelocker waits, but holds the hashbinlock. The next thread that comes in and needs the same hashbin will wait for the lock while holding the hashtable lock. thus halting the entire system on hashtable. This is because of the delete protection. Readlocks will be easier on the rwlock on entries. While the writer is holding writelock, similar problems happen with a reader or writer needing the same item. the scenario requires more than three threads. o so the queue length is 3 threads in a bad situation. The fourth is unable to use the hashtable.

If you need to acquire locks on multiple items from the hashtable. o you MUST release all locks on items from the hashtable before doing the next lookup/insert/delete/whatever. o To acquire multiple items you should use a special routine that obtains the locks on those multiple items in one go.

## Typedef Documentation

## ◆ lruhash_sizefunc_type

typedef size_t(* lruhash_sizefunc_type) (void *, void *)

Type of function that calculates the size of an entry.

Result must include the size of struct lruhash_entry. Keys that are identical must also calculate to the same size. size = func(key, data).

## ◆ lruhash_compfunc_type

typedef int(* lruhash_compfunc_type) (void *, void *)

type of function that compares two keys.

return 0 if equal.

## ◆ lruhash_delkeyfunc_type

typedef void(* lruhash_delkeyfunc_type) (void *, void *)

old keys are deleted.

The RRset type has to revoke its ID number, markdel() is used first. This function is called: func(key, userarg)

## ◆ lruhash_deldatafunc_type

typedef void(* lruhash_deldatafunc_type) (void *, void *)

old data is deleted.

This function is called: func(data, userarg).

## ◆ lruhash_markdelfunc_type

typedef void(* lruhash_markdelfunc_type) (void *)

mark a key as pending to be deleted (and not to be used by anyone).

called: func(key)

## Function Documentation

## ◆ lruhash_create()

struct lruhash * lruhash_create | ( | size_t | start_size,
|  | size_t | maxmem,
|  | lruhash_sizefunc_type | sizefunc,
|  | lruhash_compfunc_type | compfunc,
|  | lruhash_delkeyfunc_type | delkeyfunc,
|  | lruhash_deldatafunc_type | deldatafunc,
|  | void * | arg )

Create new hash table.

Parameters

  start_size | size of hashtable array at start, must be power of 2.
  maxmem | maximum amount of memory this table is allowed to use.
  sizefunc | calculates memory usage of entries.
  compfunc | compares entries, 0 on equality.
  delkeyfunc | deletes key. Calling both delkey and deldata will also free the struct lruhash_entry. Make it part of the key structure and delete it in delkeyfunc.
  deldatafunc | deletes data.
  arg | user argument that is passed to user function calls.

Returns
  : new hash table or NULL on malloc failure.

References lruhash::array, bin_init(), lruhash::cb_arg, lruhash::compfunc, lruhash::deldatafunc, lruhash::delkeyfunc, lruhash::lock, lruhash::lru_end, lruhash::lru_start, lruhash::max_collisions, lruhash::num, lruhash::size, lruhash::size_mask, lruhash::sizefunc, lruhash::space_max, and lruhash::space_used.

Referenced by lruhash_test(), and slabhash_create().

## ◆ lruhash_delete()

void lruhash_delete | ( | struct lruhash * | table | ) |

Delete hash table.

Entries are all deleted.

Parameters

  table | to delete.

References lruhash::array, bin_delete(), lruhash::lock, and lruhash::size.

Referenced by lruhash_test(), and slabhash_delete().

## ◆ lruhash_clear()

void lruhash_clear | ( | struct lruhash * | table | ) |

Clear hash table.

Entries are all deleted, while locking them before doing so. At end the table is empty.

Parameters

  table | to make empty.

References lruhash::array, bin_clear(), lruhash::deldatafunc, lruhash::delkeyfunc, fptr_ok, fptr_whitelist_hash_deldatafunc(), fptr_whitelist_hash_delkeyfunc(), fptr_whitelist_hash_markdelfunc(), lruhash::lock, lruhash::lru_end, lruhash::lru_start, lruhash::markdelfunc, lruhash::num, lruhash::size, and lruhash::space_used.

Referenced by slabhash_clear(), and test_long_table().

## ◆ lruhash_insert()

void lruhash_insert | ( | struct lruhash * | table,
|  | hashvalue_type | hash,
|  | struct lruhash_entry * | entry,
|  | void * | data,
|  | void * | cb_override )

Insert a new element into the hashtable.

If key is already present data pointer in that entry is updated. The space calculation function is called with the key, data. If necessary the least recently used entries are deleted to make space. If necessary the hash array is grown up.

Parameters

  table | hash table.
  hash | hash value. User calculates the hash.
  entry | identifies the entry. If key already present, this entry->key is deleted immediately. But entry->data is set to NULL before deletion, and put into the existing entry. The data is then freed.
  data | the data.
  cb_override | if not null overrides the cb_arg for the deletefunc.

References lruhash::array, bin_find_entry(), lruhash::cb_arg, lruhash::compfunc, lruhash_entry::data, lruhash::deldatafunc, lruhash::delkeyfunc, fptr_ok, fptr_whitelist_hash_compfunc(), fptr_whitelist_hash_deldatafunc(), fptr_whitelist_hash_delkeyfunc(), fptr_whitelist_hash_markdelfunc(), fptr_whitelist_hash_sizefunc(), lruhash_entry::hash, lruhash_entry::key, lruhash::lock, lruhash_bin::lock, lruhash_entry::lock, lru_front(), lru_touch(), lruhash::markdelfunc, lruhash::max_collisions, lruhash::num, lruhash_bin::overflow_list, lruhash_entry::overflow_next, reclaim_space(), lruhash::size, lruhash::size_mask, lruhash::sizefunc, lruhash::space_max, lruhash::space_used, and table_grow().

Referenced by slabhash_insert(), test_short_table(), testadd(), and testadd_unlim().

## ◆ lruhash_lookup()

struct lruhash_entry * lruhash_lookup | ( | struct lruhash * | table,
|  | hashvalue_type | hash,
|  | void * | key,
|  | int | wr )

Lookup an entry in the hashtable.

At the end of the function you hold a (read/write)lock on the entry. The LRU is updated for the entry (if found).

Parameters

  table | hash table.
  hash | hash of key.
  key | what to look for, compared against entries in overflow chain. the hash value must be set, and must work with compare function.
  wr | set to true if you desire a writelock on the entry. with a writelock you can update the data part.

Returns
  : pointer to the entry or NULL. The entry is locked. The user must unlock the entry when done.

References lruhash::array, bin_find_entry(), lruhash::compfunc, fptr_ok, fptr_whitelist_hash_compfunc(), lruhash::lock, lruhash_bin::lock, lru_touch(), and lruhash::size_mask.

Referenced by slabhash_lookup(), test_short_table(), testlookup(), and testlookup_unlim().

## ◆ lru_touch()

void lru_touch | ( | struct lruhash * | table,
|  | struct lruhash_entry * | entry )

Touch entry, so it becomes the most recently used in the LRU list.

Caller must hold hash table lock. The entry must be inserted already.

Parameters

  table | hash table.
  entry | entry to make first in LRU.

References log_assert, lru_front(), lru_remove(), and lruhash::lru_start.

Referenced by lruhash_insert(), lruhash_lookup(), and rrset_cache_touch().

## ◆ lruhash_update_space_used()

void lruhash_update_space_used | ( | struct lruhash * | table,
|  | void * | cb_override,
|  | int | diff_size )

Update the size of an element in the hashtable.

Parameters

  table | hash table.
  cb_override | if not NULL overrides the cb_arg for deletefunc.
  diff_size | difference in size to the hash table storage. This is newsize - oldsize, a positive number uses more space.

References lruhash::cb_arg, lruhash_entry::data, lruhash::deldatafunc, lruhash::delkeyfunc, fptr_ok, fptr_whitelist_hash_deldatafunc(), fptr_whitelist_hash_delkeyfunc(), fptr_whitelist_hash_markdelfunc(), fptr_whitelist_hash_sizefunc(), lruhash_entry::key, lruhash::lock, lruhash::markdelfunc, lruhash_entry::overflow_next, reclaim_space(), lruhash::sizefunc, lruhash::space_max, and lruhash::space_used.

Referenced by slabhash_update_space_used().

## ◆ lru_demote()

void lru_demote | ( | struct lruhash * | table,
|  | struct lruhash_entry * | entry )

Demote entry, so it becomes the least recently used in the LRU list.

Caller must hold hash table lock. The entry must be inserted already.

Parameters

  table | hash table.
  entry | entry to make last in LRU.

References log_assert, lruhash::lru_end, lruhash_entry::lru_next, lru_remove(), and lruhash::lru_start.

## ◆ lruhash_insert_or_retrieve()

struct lruhash_entry * lruhash_insert_or_retrieve | ( | struct lruhash * | table,
|  | hashvalue_type | hash,
|  | struct lruhash_entry * | entry,
|  | void * | data,
|  | void * | cb_arg )

Insert a new element into the hashtable, or retrieve the corresponding element of it exits.

If key is already present data pointer in that entry is kept. If it is not present, a new entry is created. In that case, the space calculation function is called with the key, data. If necessary the least recently used entries are deleted to make space. If necessary the hash array is grown up.

Parameters

  table | hash table.
  hash | hash value. User calculates the hash.
  entry | identifies the entry.
  data | the data.
  cb_arg | if not null overrides the cb_arg for the deletefunc.

Returns
  : pointer to the existing entry if the key was already present, or to the entry argument if it was not.

References lruhash::array, bin_find_entry(), lruhash::cb_arg, lruhash::compfunc, lruhash_entry::data, lruhash::deldatafunc, lruhash::delkeyfunc, fptr_ok, fptr_whitelist_hash_compfunc(), fptr_whitelist_hash_deldatafunc(), fptr_whitelist_hash_delkeyfunc(), fptr_whitelist_hash_markdelfunc(), fptr_whitelist_hash_sizefunc(), lruhash_entry::hash, lruhash_entry::key, lruhash::lock, lruhash_bin::lock, lruhash_entry::lock, lru_front(), lruhash::markdelfunc, lruhash::max_collisions, lruhash::num, lruhash_bin::overflow_list, lruhash_entry::overflow_next, reclaim_space(), lruhash::size, lruhash::size_mask, lruhash::sizefunc, lruhash::space_max, lruhash::space_used, and table_grow().

## ◆ lruhash_remove()

void lruhash_remove | ( | struct lruhash * | table,
|  | hashvalue_type | hash,
|  | void * | key )

Remove entry from hashtable.

Does nothing if not found in hashtable. Delfunc is called for the entry.

Parameters

  table | hash table.
  hash | hash of key.
  key | what to look for.

References lruhash::array, bin_find_entry(), bin_overflow_remove(), lruhash::cb_arg, lruhash::compfunc, lruhash::deldatafunc, lruhash::delkeyfunc, fptr_ok, fptr_whitelist_hash_compfunc(), fptr_whitelist_hash_deldatafunc(), fptr_whitelist_hash_delkeyfunc(), fptr_whitelist_hash_markdelfunc(), fptr_whitelist_hash_sizefunc(), lruhash::lock, lruhash_bin::lock, lru_remove(), lruhash::markdelfunc, lruhash::num, lruhash::size_mask, lruhash::sizefunc, and lruhash::space_used.

Referenced by slabhash_remove(), test_short_table(), testremove(), and testremove_unlim().

## ◆ bin_find_entry()

struct lruhash_entry * bin_find_entry | ( | struct lruhash * | table,
|  | struct lruhash_bin * | bin,
|  | hashvalue_type | hash,
|  | void * | key,
|  | size_t * | collisions )

Find entry in hash bin.

You must have locked the bin.

Parameters

  table | hash table with function pointers.
  bin | hash bin to look into.
  hash | hash value to look for.
  key | key to look for.
  collisions | how many collisions were found during the search.

Returns
  : the entry or NULL if not found.

References lruhash::compfunc, lruhash_entry::hash, lruhash_entry::key, lruhash_bin::overflow_list, and lruhash_entry::overflow_next.

Referenced by lruhash_insert(), lruhash_insert_or_retrieve(), lruhash_lookup(), lruhash_remove(), and test_bin_find_entry().

## ◆ bin_overflow_remove()

void bin_overflow_remove | ( | struct lruhash_bin * | bin,
|  | struct lruhash_entry * | entry )

Remove entry from bin overflow chain.

You must have locked the bin.

Parameters

  bin | hash bin to look into.
  entry | entry ptr that needs removal.

References lruhash_bin::overflow_list, and lruhash_entry::overflow_next.

Referenced by lruhash_remove(), reclaim_space(), and test_bin_find_entry().

## ◆ bin_split()

void bin_split | ( | struct lruhash * | table,
|  | struct lruhash_bin * | newa,
|  | int | newmask )

Split hash bin into two new ones.

Based on increased size_mask. Caller must hold hash table lock. At the end the routine acquires all hashbin locks (in the old array). This makes it wait for other threads to finish with the bins. So the bins are ready to be deleted after this function.

Parameters

  table | hash table with function pointers.
  newa | new increased array.
  newmask | new lookup mask.

References lruhash::array, lruhash_entry::hash, lruhash_bin::lock, lruhash_bin::overflow_list, lruhash_entry::overflow_next, lruhash::size, and lruhash::size_mask.

Referenced by table_grow().

## ◆ reclaim_space()

void reclaim_space | ( | struct lruhash * | table,
|  | struct lruhash_entry ** | list )

Try to make space available by deleting old entries.

Assumes that the lock on the hashtable is being held by caller. Caller must not hold bin locks.

Parameters

  table | hash table.
  list | list of entries that are to be deleted later. Entries have been removed from the hash table and writelock is held.

References lruhash::array, bin_overflow_remove(), lruhash_entry::data, lruhash_entry::hash, lruhash_entry::key, lruhash_bin::lock, lruhash_entry::lock, log_assert, lruhash::lru_end, lruhash_entry::lru_next, lruhash_entry::lru_prev, lruhash::markdelfunc, lruhash::num, lruhash_entry::overflow_next, lruhash::size_mask, lruhash::sizefunc, lruhash::space_max, and lruhash::space_used.

Referenced by lruhash_insert(), lruhash_insert_or_retrieve(), and lruhash_update_space_used().

## ◆ table_grow()

void table_grow | ( | struct lruhash * | table | ) |

Grow the table lookup array.

Becomes twice as large. Caller must hold the hash table lock. Must not hold any bin locks. Tries to grow, on malloc failure, nothing happened.

Parameters

  table | hash table.

References lruhash::array, bin_init(), bin_split(), lruhash::lock, lruhash_bin::lock, log_err(), lruhash::size, and lruhash::size_mask.

Referenced by lruhash_insert(), and lruhash_insert_or_retrieve().

## ◆ lru_front()

void lru_front | ( | struct lruhash * | table,
|  | struct lruhash_entry * | entry )

Put entry at front of lru.

entry must be unlinked from lru. Caller must hold hash table lock.

Parameters

  table | hash table with lru head and tail.
  entry | entry to make most recently used.

References lruhash::lru_end, lruhash_entry::lru_prev, and lruhash::lru_start.

Referenced by lru_touch(), lruhash_insert(), lruhash_insert_or_retrieve(), and test_lru().

## ◆ lru_remove()

void lru_remove | ( | struct lruhash * | table,
|  | struct lruhash_entry * | entry )

Remove entry from lru list.

Caller must hold hash table lock.

Parameters

  table | hash table with lru head and tail.
  entry | entry to remove from lru.

References lruhash::lru_end, and lruhash::lru_start.

Referenced by lru_demote(), lru_touch(), lruhash_remove(), and test_lru().

## ◆ lruhash_status()

void lruhash_status | ( | struct lruhash * | table,
|  | const char * | id,
|  | int | extended )

Output debug info to the log as to state of the hash table.

Parameters

  table | hash table.
  id | string printed with table to identify the hash table.
  extended | set to true to print statistics on overflow bin lengths.

References lruhash::array, lruhash::lock, lruhash_bin::lock, log_info(), lruhash::num, lruhash_bin::overflow_list, lruhash_entry::overflow_next, lruhash::size, lruhash::size_mask, lruhash::space_max, and lruhash::space_used.

Referenced by slabhash_status(), test_long_table(), test_thr_main(), and test_threaded_table().

## ◆ lruhash_get_mem()

size_t lruhash_get_mem | ( | struct lruhash * | table | ) |

Get memory in use now by the lruhash table.

Parameters

  table | hash table. Will be locked before use. And unlocked after.

Returns
  size in bytes.

References lruhash::array, lruhash::lock, lruhash_bin::lock, lruhash::size, and lruhash::space_used.

Referenced by slabhash_get_mem().

## ◆ lruhash_traverse()

void lruhash_traverse | ( | struct lruhash * | h,
|  | int | wr,
|  | void(* | func )(struct lruhash_entry *, void *),
|  | void * | arg )

Traverse a lruhash.

Call back for every element in the table.

Parameters

  h | hash table. Locked before use.
  wr | if true writelock is obtained on element, otherwise readlock.
  func | function for every element. Do not lock or unlock elements.
  arg | user argument to func.

References lruhash::array, lruhash::lock, lruhash_bin::lock, lruhash_entry::lock, lruhash_bin::overflow_list, lruhash_entry::overflow_next, and lruhash::size.

Referenced by slabhash_traverse().