hazptr: Improve documentation

Summary: Improve documentation and make the description of synchronous reclamation  consistent with the current implementation.

Reviewed By: yfeldblum

Differential Revision: D28575528

fbshipit-source-id: e8f37e02d18b12a0e653264c128492a98cba6a1d

folly/synchronization/Hazptr-fwd.h

@@ -47,7 +47,7 @@ template <template <typename> class Atom = std::atomic>
 class hazptr_rec;
 
 ///
-/// Classes related to objects protected by hazard pointers.
+/// Classes related to objects protectable by hazard pointers.
 /// Defined in HazptrObj.h
 ///
 

folly/synchronization/Hazptr.h

@@ -187,17 +187,14 @@
 /// - Both can protect linked structures. Hazard pointers needs
 ///   additional link counting with low or moderate overhead for
 ///   update operations, and no overhead for readers. RCU protects
-///   protects linked structures automatically, because it protects
-///   everything.
+///   linked structures automatically, because it implicitly protects
+///   all protectable objects.
 ///
 /// Differences from the Standard Proposal
 /// --------------------------------------
-/// - The latest standard proposal is in wg21.link/p0566.
-/// - This library's API differs from the standard proposal because:
-///   (a) the standard proposal is changing based on committee
-///   feedback, and (b) this library provides additional
-///   fast-evolving features based on usage experience that do not
-///   have corressponding proposed standard wording.
+/// - The latest standard proposal is in wg21.link/p1121. The
+///   substance of the proposal was frozen around October 2017, but
+///   there were subsequent API changes based on committee feadback.
 /// - The main differences are:
 ///   o This library uses an extra atomic template parameter for
 ///     testing and debugging.
@@ -213,3 +210,5 @@
 ///     proposal.
 ///   o Link counting support and protection of linked structures is
 ///     not part of the current standard proposal.
+///   o The standard proposal does not include cohorts and the
+///     associated synchronous reclamation capabilities.

folly/synchronization/HazptrDomain.h

@@ -17,7 +17,7 @@
 #pragma once
 
 #include <atomic>
-#include <unordered_set> // for hash set in bulk_reclaim
+#include <unordered_set>
 
 #include <folly/Memory.h>
 #include <folly/Portability.h>
@@ -36,6 +36,8 @@ namespace folly {
 
 namespace detail {
 
+/** Threshold for the number of retired objects to trigger
+    asynchronous reclamation. */
 constexpr int hazptr_domain_rcount_threshold() {
   return 1000;
 }
@@ -49,58 +51,46 @@ constexpr int hazptr_domain_rcount_threshold() {
  *
  *  Most user code need not specify any domains.
  *
- *  Notes on destruction order, tagged objects, locking and deadlock
- *  avoidance:
- *  - Tagged objects support reclamation order guarantees. A call to
- *    cleanup_cohort_tag(tag) guarantees that all objects with the
- *    specified tag are reclaimed before the function returns.
- *  - Due to the strict order, access to the set of tagged objects
- *    needs synchronization and care must be taken to avoid deadlock.
+ *  Notes on destruction order and tagged objects:
+ *  - Tagged objects support reclamation order guarantees (i.e.,
+ *    synchronous reclamation). A call to cleanup_cohort_tag(tag)
+ *    guarantees that all objects with the specified tag are reclaimed
+ *    before the function returns.
  *  - There are two types of reclamation operations to consider:
- *   - Type A: A Type A reclamation operation is triggered by meeting
- *     some threshold. Reclaimed objects may have different
- *     tags. Hazard pointers are checked and only unprotected objects
- *     are reclaimed. This type is expected to be expensive but
- *     infrequent and the cost is amortized over a large number of
- *     reclaimed objects. This type is needed to guarantee an upper
- *     bound on unreclaimed reclaimable objects.
- *   - Type B: A Type B reclamation operation is triggered by a call
- *     to the function cleanup_cohort_tag for a specific tag. All
- *     objects with the specified tag must be reclaimed
- *     unconditionally before returning from such a function
- *     call. Hazard pointers are not checked. This type of reclamation
- *     operation is expected to be inexpensive and may be invoked more
- *     frequently than Type A.
- *  - Tagged retired objects are kept in a single list in the domain
- *    structure, named tagged_.
- *  - Both Type A and Type B of reclamation pop all the objects in
- *    tagged_ and sort them into two sets of reclaimable and
- *    unreclaimable objects. The objects in the reclaimable set are
- *    reclaimed and the objects in the unreclaimable set are pushed
- *    back in tagged_.
- *  - The tagged_ list is locked between popping all objects and
+ *   - Asynchronous reclamation: It is triggered by meeting some
+ *     threshold for the number of retired objects or the time since
+ *     the last asynchronous reclamation. Reclaimed objects may have
+ *     different tags or no tags. Hazard pointers are checked and only
+ *     unprotected objects are reclaimed. This type is expected to be
+ *     expensive but infrequent and the cost is amortized over a large
+ *     number of reclaimed objects. This type is needed to guarantee
+ *     an upper bound on unreclaimed reclaimable objects.
+ *   - Synchronous reclamation: It is invoked by calling
+ *     cleanup_cohort_tag for a specific tag. All objects with the
+ *     specified tag must be reclaimed unconditionally before
+ *     returning from such a function call. Hazard pointers are not
+ *     checked. This type of reclamation operation is expected to be
+ *     inexpensive and may be invoked more frequently than
+ *     asynchronous reclamation.
+ *  - Tagged retired objects are kept in a sharded list in the domain
+ *    structure.
+ *  - Both asynchronous and synchronous reclamation pop all the
+ *    objects in the tagged list(s) and sort them into two sets of
+ *    reclaimable and unreclaimable objects. The objects in the
+ *    reclaimable set are reclaimed and the objects in the
+ *    unreclaimable set are pushed back in the tagged list(s).
+ *  - The tagged list(s) are locked between popping all objects and
  *    pushing back unreclaimable objects, in order to guarantee that
- *    Type B operations do not miss any objects that match the
- *    specified tag.
- *  - A Type A operation cannot release the lock on the tagged_ list
- *    before reclaiming reclaimable objects, to prevent concurrent
- *    Type B operations from returning before the reclamation of
- *    objects with matching tags.
- *  - A Type B operation can release the lock on tagged_ before
- *    reclaiming objects because the set of reclaimable objects by
- *    Type B operations are disjoint.
- *  - The lock on the tagged_ list is re-entrant, to prevent deadlock
- *    when reclamation in a Type A operation requires a Type B
- *    reclamation operation to complete.
- *  - The implementation allows only one pattern of re-entrance: An
- *    inner Type B inside an outer Type A.
- *  - An inner Type B operation must have access and ability to modify
- *    the outer Type A operation's set of reclaimable objects and
- *    their children objects in order not to miss objects that match
- *    the specified tag. Hence, Type A operations use data members,
- *    unprotected_ and children_, to keep track of these objects
- *    between reclamation steps and to provide inner Type B operations
- *    access to these objects.
+ *    synchronous reclamation operations do not miss any objects.
+ *  - Asynchronous reclamation can release the lock(s) on the tagged
+ *    list(s) before reclaiming reclaimable objects, because it pushes
+ *    reclaimable tagged objects in their respective cohorts, which
+ *    would handle concurrent synchronous reclamation of such objects
+ *    properly.
+ *  - Synchronous reclamation operations can release the lock on the
+ *    tagged list shard before reclaiming objects because the sets of
+ *    reclaimable objects by different synchronous reclamation
+ *    operations are disjoint.
  */
 template <template <typename> class Atom>
 class hazptr_domain {
@@ -563,7 +553,7 @@ class hazptr_domain {
 
   void free_hazptr_recs() {
     /* Leak the hazard pointers for the default domain to avoid
-       destruction order issues with thread caches.  */
+       destruction order issues with thread caches. */
     if (this == &default_hazptr_domain<Atom>()) {
       return;
     }
@@ -632,7 +622,7 @@ class hazptr_domain {
       /*** Full fence ***/ asymmetricHeavyBarrier(AMBFlags::EXPEDITED);
       auto rec = hazptrs_.load(std::memory_order_acquire);
       /* Part 1 - read hazard pointer values into private search structure */
-      std::unordered_set<const void*> hashset; // TOTO: lock-free fixed hash set
+      std::unordered_set<const void*> hashset;
       for (; rec; rec = rec->next()) {
         hashset.insert(rec->hazptr());
       }

folly/synchronization/HazptrObj.h

@@ -28,22 +28,22 @@
 #include <folly/synchronization/detail/HazptrUtils.h>
 
 ///
-/// Classes related to objects protected by hazard pointers.
+/// Classes related to objects protectable by hazard pointers.
 ///
 
 namespace folly {
 
 /**
  *  hazptr_obj
  *
- *  Private base class for objects protected by hazard pointers.
+ *  Private base class for objects protectable by hazard pointers.
  *
  *  Data members:
  *  - next_: link to next object in private singly linked lists.
  *  - reclaim_: reclamation function for this object.
- *  - cohort_tag_: A pointer to a cohort (a linked list where the
- *    object is to be pushed when retired). It can also be used as a
- *    tag (see below). See details below.
+ *  - cohort_tag_: A pointer to a cohort (where the object is to be
+ *    pushed when retired). It can also be used as a tag (see
+ *    below).
  *
  *  Cohorts, Tags, Tagged Objects, and Untagged Objects:
  *
@@ -53,27 +53,27 @@ namespace folly {
  *    and/or mixed with unrelated objects.
  *
  *  - Tags: A tag is a unique identifier used for fast identification
- *    of related objects. Tags are implemented as addresses of
- *    cohorts, with the lowest bit set (to save the space of separate
- *    cohort and tag data members and to differentiate from cohorts of
- *    untagged objects.
+ *    of related objects for synchronous reclamation. Tags are
+ *    implemented as addresses of cohorts, with the lowest bit set (to
+ *    save the space of separate cohort and tag data members and to
+ *    differentiate from cohorts of untagged objects.
  *
  *  - Tagged objects: Objects are tagged for fast identification. The
- *    primary use case is for guaranteeing the destruction of all
- *    objects with a certain tag (e.g., the destruction of all Key and
- *    Value objects that were part of a Folly ConcurrentHashMap
- *    instance). Member function set_cohort_tag makes an object tagged.
+ *    primary use case is for synchronous reclamation ((e.g., the
+ *    destruction of all Key and Value objects that were part of a
+ *    Folly ConcurrentHashMap instance). Member function
+ *    set_cohort_tag makes an object tagged.
  *
  *  - Untagged objects: Objects that do not need to be identified
  *    separately from unrelated objects are not tagged (to keep tagged
  *    objects uncluttered). Untagged objects may or may not be
  *    associated with cohorts. An example of untagged objects
  *    associated with cohorts are Segment-s of Folly UnboundedQueue.
- *    Although such objects do not need to be tagged, keeping them in
- *    cohorts helps avoid cases of a few missing objects delaying the
- *    reclamation of large numbers of link-counted objects. Objects
- *    are untagged either by default or after calling
- *    set_cohort_no_tag.
+ *    Although such objects do not need synchronous reclamation,
+ *    keeping them in cohorts helps avoid cases of a few missing
+ *    objects delaying the reclamation of large numbers of
+ *    link-counted objects. Objects are untagged either by default or
+ *    after calling set_cohort_no_tag.
  *
  *  - Thread Safety: Member functions set_cohort_tag and
  *    set_cohort_no_tag are not thread-safe. Thread safety must be

folly/synchronization/detail/HazptrUtils.h

@@ -25,7 +25,7 @@
 #include <folly/synchronization/detail/Sleeper.h>
 
 /// Linked list class templates used in the hazard pointer library:
-/// - linked_list: Sequential linked list that uses a pre-existing
+/// - linked_list: Sequential linked list that uses pre-existing
 ///   members next() and set_next();.
 /// - shared_head_tail_list: Thread-safe linked list that maintains
 ///   head and tail pointers. Supports push and pop_all.