Command Interface

The RMR command interface provides upper layer clients (e.g. BRMR) a way to send command messages to all storage nodes of an RMR pool, and gives the storage-side handler a way to send a response back. The size of the per-leg response buffer (size) is specified by the caller, making it dynamic.

The interface is exposed through rmr_clt_cmd_with_rsp() on the client side and rmr_srv_pool_cmd_with_rsp() on the server side.

Buffer allocation

At any point in time the caller does not know how many legs the RMR pool has. Querying first and then calling would create a race window during which the number of legs can change — particularly in setups where legs are added or removed frequently.

The solution is to always allocate for the maximum number of legs (RMR_POOL_MAX_SESS, currently 4). The caller allocates a single physically contiguous buffer (kmalloc’d — vmalloc memory is rejected) of size RMR_POOL_MAX_SESS * size. This is a hard requirement: rmr_clt_cmd_with_rsp() returns -EINVAL if buf_len != RMR_POOL_MAX_SESS * size.

RMR divides the buffer into RMR_POOL_MAX_SESS slices of size bytes, maps each to an sglist, and sends one to each leg. If the pool has fewer than RMR_POOL_MAX_SESS legs, the leftover slices are untouched. The server-side handler writes its response into the slice it received. The caller must have a way to distinguish written slices from untouched ones.

The command message itself is passed as an array of kvecs and forwarded to the server side as-is.

Interaction with map updates

In-flight command requests block map updates until they complete. Callers should be aware that issuing commands during recovery may delay the map update process.

Sessions that receive commands

Commands are sent to all sessions that are not in FAILED state (CREATED, NORMAL, RECONNECTING, and REMOVING sessions all accept command messages). FAILED sessions are skipped silently, so the caller may receive fewer responses than the total number of legs.

Error handling

Error on user level

The server-side handler may fail to execute the command. In this case the error should be encoded in the response buffer and not propagated to RMR. The caller reads it while processing the response slices.

Error on RMR level

If the command fails in the RMR or RTRS layer (e.g. network failure, memory allocation), the result is reported via the conf callback (rmr_conf_fn). The errno passed to conf has the following encoding:

  • 0 — command sent and response received from all legs successfully.

  • positive — command failed on some legs. The value is the number of failed legs.

  • negative — command failed on all legs. The value is the error code.