aihwkit.simulator.configs.devices module

Configuration for Analog (Resistive Device) tiles.

class aihwkit.simulator.configs.devices.ConstantStepDevice(construction_seed=0, corrupt_devices_prob=0.0, corrupt_devices_range=1000, diffusion=0.0, diffusion_dtod=0.0, dw_min=0.001, dw_min_dtod=0.3, dw_min_std=0.3, enforce_consistency=True, lifetime=0.0, lifetime_dtod=0.0, perfect_bias=False, reset=0.01, reset_dtod=0.0, reset_std=0.01, up_down=0.0, up_down_dtod=0.01, w_max=0.6, w_max_dtod=0.3, w_min=- 0.6, w_min_dtod=0.3)

Bases: aihwkit.simulator.configs.devices.PulsedDevice

Pulsed update behavioral model: constant step.

Pulsed update behavioral model, where the update step of material is constant throughout the resistive range (up to hard bounds).

In more detail, the update behavior implemented for ConstantStep is:

\[ \begin{align}\begin{aligned}w_{ij} &\leftarrow& w_{ij} - \Delta w_{ij}^d(1 + \sigma_\text{c-to-c}\,\xi)\\w_{ij} &\leftarrow& \text{clip}(w_{ij},b^\text{min}_{ij},b^\text{max}_{ij})\end{aligned}\end{align} \]

where \(d\) is the direction of the update (product of signs of input and error). \(\Delta w_{ij}^d\) is the update step size of the cross-point ij in direction \(d\) (up or down). Note that each cross-point has separate update sizes so that device-to-device fluctuations and biases in the directions can be given.

Moreover, the clipping bounds of each cross-point ij (i.e. \(b_{ij}^\text{max/min}\)) are also different in general. The mean and the amount of systematic spread from device-to-device can be given as parameters, see below.

For parameters regarding the devices settings, see e.g. ConstantStepResistiveDeviceParameters.

class aihwkit.simulator.configs.devices.DifferenceUnitCellDevice(unit_cell_devices=<factory>)

Bases: aihwkit.simulator.configs.devices.UnitCellDevice

Abstract device model takes an arbitrary device per crosspoint and implements an explicit plus-minus device pair.

A plus minus pair is implemented by using only one-sided updated of the given devices. Note that reset might need to be called otherwise the one-sided device quickly saturates during learning.

The output current is the difference of both devices.

Meta parameter setting of the pairs are assumed to be identical (however, device-to-device variation is still present).

Caution

Reset needs to be added manually by calling the reset_columns method of a tile.

as_bindings()

Return a representation of this instance as a simulator bindings object.

Return type

aihwkit.simulator.rpu_base.devices.DifferenceResistiveDeviceParameter

class aihwkit.simulator.configs.devices.ExpStepDevice(construction_seed=0, corrupt_devices_prob=0.0, corrupt_devices_range=1000, diffusion=0.0, diffusion_dtod=0.0, dw_min=0.001, dw_min_dtod=0.3, dw_min_std=0.3, enforce_consistency=True, lifetime=0.0, lifetime_dtod=0.0, perfect_bias=False, reset=0.01, reset_dtod=0.0, reset_std=0.01, up_down=0.0, up_down_dtod=0.01, w_max=0.6, w_max_dtod=0.3, w_min=- 0.6, w_min_dtod=0.3, A_up=0.00081, A_down=0.36833, gamma_up=12.44625, gamma_down=12.78785, a=0.244, b=0.2425)

Bases: aihwkit.simulator.configs.devices.PulsedDevice

Exponential update step or CMOS-like update behavior.

This model is derived from PulsedResistiveDevice and uses all its parameters. ExpStepResistiveDevice only implements a new ‘update once’ functionality, where the minimal weight step change with weight is fitted by an exponential function as detailed below.

\[w_{ij} \leftarrow w_{ij} - \max(y_{ij},0) \Delta w_{ij}^d (1 + \sigma_\text{c-to-c}\,\xi)\]

and \(y_{ij}\) is given as

\[ \begin{align}\begin{aligned}z_{ij} = 2 a_\text{es} \frac{w_{ij}}{b^\text{max}_{ij} - b^\text{min}_{ij}} + b_\text{es}\\y_{ij} = 1 - A^{(d)} e^{d \gamma^{(d)} z_{ij}}\end{aligned}\end{align} \]

where \(d\) is the direction of the update (+ or -), see also ConstantStepResistiveDevice for details.

All additional parameter (\(a_\text{es}\), \(b_\text{es}\), \(\gamma^{(d)}\), \(A^{(d)}\) ) are tile-wise fitting parameters (ie. no device-to-device variation in these parameters). Note that the other parameter involved can be still defined with device-to-device variation and (additional) up-down bias (see ConstantStepResistiveDevice).

A_down: float = 0.36833

Factor A for the down direction

A_up: float = 0.00081

Factor A for the up direction

a: float = 0.244

Global slope parameter

b: float = 0.2425

Global offset parameter

gamma_down: float = 12.78785

Exponent for the down direction.

gamma_up: float = 12.44625

Exponent for the up direction.

class aihwkit.simulator.configs.devices.FloatingPointDevice(diffusion=0.0, lifetime=0.0)

Bases: object

Floating point reference.

Implements ideal devices forward/backward/update behavior.

as_bindings()

Return a representation of this instance as a simulator bindings object.

Return type

aihwkit.simulator.rpu_base.devices.FloatingPointTileParameter

diffusion: float = 0.0

Standard deviation of diffusion process.

lifetime: float = 0.0

One over decay_rate, ie \(1/r_\text{decay}\).

class aihwkit.simulator.configs.devices.IdealDevice(construction_seed=0, diffusion=0.0, lifetime=0.0)

Bases: object

Ideal update behavior (using floating point), but forward/backward might be non-ideal.

Ideal update behavior (using floating point), however, forward/backward might still have a non-ideal ADC or noise added.

as_bindings()

Return a representation of this instance as a simulator bindings object.

Return type

aihwkit.simulator.rpu_base.devices.IdealResistiveDeviceParameter

construction_seed: int = 0

If not equal 0, will set a unique seed for hidden parameters during construction

diffusion: float = 0.0

Standard deviation of diffusion process.

lifetime: float = 0.0

One over decay_rate, ie \(1/r_\text{decay}\).

class aihwkit.simulator.configs.devices.LinearStepDevice(construction_seed=0, corrupt_devices_prob=0.0, corrupt_devices_range=1000, diffusion=0.0, diffusion_dtod=0.0, dw_min=0.001, dw_min_dtod=0.3, dw_min_std=0.3, enforce_consistency=True, lifetime=0.0, lifetime_dtod=0.0, perfect_bias=False, reset=0.01, reset_dtod=0.0, reset_std=0.01, up_down=0.0, up_down_dtod=0.01, w_max=0.6, w_max_dtod=0.3, w_min=- 0.6, w_min_dtod=0.3, gamma_up=0.0, gamma_down=0.0, gamma_up_dtod=0.05, gamma_down_dtod=0.05, allow_increasing=False, mean_bound_reference=True, mult_noise=True)

Bases: aihwkit.simulator.configs.devices.PulsedDevice

Pulsed update behavioral model: linear step.

Pulsed update behavioral model, where the update step response size of the material is linearly dependent with resistance (up to hard bounds).

This model is very similar to ConstantStepResistiveDevice and thus shares all parameters and functionality. In addition, it only implements a more general update once function, where the update step size can depend linearly on the weight itself.

For each coincidence the weights is updated once. Here, the positive (negative) update step size decreases linearly in the following manner (compare to the update once for ConstantStepResistiveDevice):

\begin{eqnarray*} w_{ij} &\leftarrow& w_{ij} - \Delta w_{ij}^d(\gamma_{ij}^d\;w_{ij} + 1 + \sigma_\text{c-to-c}\,\xi)\\ w_{ij} &\leftarrow& \text{clip}(w_{ij},b^\text{min}_{ij},b^\text{max}_{ij}) \end{eqnarray*}

in case of additive noise. Optionally, multiplicative noise can be chosen in which case the first equation becomes:

\[w_{ij} \leftarrow w_{ij} - \Delta w_{ij}^d (\gamma_{ij}^d \;w_{ij} + 1) (1 + \sigma_\text{c-to-c}\,\xi)\]

The cross-point ij dependent slope parameter \(\gamma_{ij}^d\) are given during initialization by

\begin{eqnarray*} \gamma_{ij}^+ &=& - |\gamma^+ + \gamma_\text{d-to-d}^+ \xi|/b^\text{max}_{ij}\\ \gamma_{ij}^- &=& - |\gamma^- + \gamma_\text{d-to-d}^- \xi|/b^\text{min}_{ij} \end{eqnarray*}

where the \(\xi\) are standard Gaussian random variables and \(b^\text{min}_{ij}\) and \(b^\text{max}_{ij}\) the cross-point ij specific minimal and maximal weight bounds, respectively (see description for ConstantStepResistiveDevice).

Note

If \(\gamma=1\) and \(\gamma_\text{d-to-d}=0\) this update implements soft bounds, since the updates step becomes equal to \(1/b\).

Note

If \(\gamma=0\) and \(\gamma_\text{d-to-d}=0\) and additive noise, this update is identical to ConstantStepResistiveDevice.

allow_increasing: bool = False

Whether to allow the situation where update sizes increase towards the bound instead of saturating (and thus becoming smaller)

gamma_down: float = 0.0

The value of \(\gamma^-\).

gamma_down_dtod: float = 0.05

Device-to-device variation for \(\gamma^-\), i.e. the value of \(\gamma_\text{d-to-d}^-\).

gamma_up: float = 0.0

The value of \(\gamma^+\).

Intuitively, a value of 0.1 means that the update step size in up direction at the weight bounds is 10% decreased relative to that origin \(w=0\).

Note

In principle one could fix \(\gamma=\gamma^-=\gamma^+\) since up/down variation can be given by up_down_dtod, see ConstantStepResistiveDevice.

Note

The hard-bounds are still observed, so that the weight cannot grow beyond its bounds.

gamma_up_dtod: float = 0.05

Device-to-device variation for \(\gamma^+\), i.e. the value of \(\gamma_\text{d-to-d}^+\).

mean_bound_reference: bool = True

Whether to use instead of the above:

\[ \begin{align}\begin{aligned}\gamma_{ij}^+ &=& - |\gamma^+ + \gamma_\text{d-to-d}^+ \xi|/b^\text{max}\\\gamma_{ij}^- &=& - |\gamma^- + \gamma_\text{d-to-d}^- \xi|/b^\text{min}\end{aligned}\end{align} \]

where \(b^\text{max}\) and \(b^\text{max}\) are the values given by w_max and w_min, see ConstantStepResistiveDevice.

mult_noise: bool = True

Whether to use multiplicative noise instead of additive cycle-to-cycle noise

class aihwkit.simulator.configs.devices.PulsedDevice(construction_seed=0, corrupt_devices_prob=0.0, corrupt_devices_range=1000, diffusion=0.0, diffusion_dtod=0.0, dw_min=0.001, dw_min_dtod=0.3, dw_min_std=0.3, enforce_consistency=True, lifetime=0.0, lifetime_dtod=0.0, perfect_bias=False, reset=0.01, reset_dtod=0.0, reset_std=0.01, up_down=0.0, up_down_dtod=0.01, w_max=0.6, w_max_dtod=0.3, w_min=- 0.6, w_min_dtod=0.3)

Bases: object

Pulsed update resistive devices.

Device are used as part of an AnalogTile to implement the update once characteristics, i.e. the material response properties when a single update pulse is given (a coincidence between row and column pulse train happened).

Common properties of all pulsed devices include:

Reset:

Resets the weight in cross points to (around) zero with cycle-to-cycle and systematic spread around a mean.

Decay:

\[w_{ij} \leftarrow w_{ij}\,(1-\alpha_\text{decay}\delta_{ij})\]

Weight decay is generally off and has to be activated explicitly by using decay() on an analog tile. Note that the device decay_lifetime parameters (1 over decay rates \(\delta_{ij}\)) are analog tile specific and are thus set and fixed during RPU initialization. \(\alpha_\text{decay}\) is a scaling factor that can be given during run-time.

Diffusion:

Similar to the decay, diffusion is only activated by inserting a specific operator. However, the parameters of the diffusion process are set during RPU initialization and are fixed for the remainder.

\[w_{ij} \leftarrow w_{ij} + \rho_{ij} \, \xi;\]

where \(xi\) is a standard Gaussian variable and \(\rho_{ij}\) the diffusion rate for a cross-point ij

Note

If diffusion happens to move the weight beyond the hard bounds of the weight it is ensured to be clipped appropriately.

as_bindings()

Return a representation of this instance as a simulator bindings object.

Return type

aihwkit.simulator.rpu_base.devices.PulsedResistiveDeviceParameter

construction_seed: int = 0

If not equal 0, will set a unique seed for hidden parameters during construction

corrupt_devices_prob: float = 0.0

Probability for devices to be corrupt (weights fixed to random value with hard bounds, that is min and max bounds are set to equal).

corrupt_devices_range: int = 1000

Range around zero for establishing corrupt devices.

diffusion: float = 0.0

Standard deviation of diffusion process.

diffusion_dtod: float = 0.0

Device-to device variation of diffusion rate in relative units.

dw_min: float = 0.001

Mean of the minimal update step sizes across devices and directions.

dw_min_dtod: float = 0.3

Device-to-device std deviation of dw_min (in relative units to dw_min).

dw_min_std: float = 0.3

Cycle-to-cycle variation size of the update step (related to \(\sigma_\text{c-to-c}\) above) in relative units to dw_min.

Note

Many spread (device-to-device variation) parameters are given in relative units. For instance e.g. a setting of dw_min_std of 0.1 would mean 10% spread around the mean and thus a resulting standard deviation (\(\sigma_\text{c-to-c}\)) of dw_min * dw_min_std.

enforce_consistency: bool = True

Whether to enforce during initialization that max weight bounds cannot be smaller than min weight bounds, and up direction step size is positive and down negative. Switches the opposite values if encountered during init.

lifetime: float = 0.0

One over decay_rate, ie \(1/r_\text{decay}\).

lifetime_dtod: float = 0.0

Device-to-device variation in the decay rate (in relative units).

perfect_bias: bool = False

No up-down differences and device-to-device variability in the bounds for the devices in the bias row.

reset: float = 0.01

The reset values and spread per cross-point ij when using reset functionality of the device.

reset_dtod: float = 0.0

See reset.

reset_std: float = 0.01

See reset.

up_down: float = 0.0

Up and down direction step sizes can be systematically different and also vary across devices. \(\Delta w_{ij}^d\) is set during RPU initialization (for each cross-point ij):

\[\Delta w_{ij}^d = d\; \Delta w_\text{min}\, \left( 1 + d \beta_{ij} + \sigma_\text{d-to-d}\xi\right)\]

where xi is again a standard Gaussian. \(\beta_{ij}\) is the directional up versus down bias. At initialization up_down_dtod and up_down defines this bias term:

\[\beta_{ij} = \beta_\text{up-down} + \xi \sigma_\text{up-down-dtod}\]

where xi is again a standard Gaussian number and \(\beta_\text{up-down}\) corresponds to up_down. Note that up_down_dtod is again given in relative units to dw_min.

up_down_dtod: float = 0.01

See up_down.

w_max: float = 0.6

See w_min.

w_max_dtod: float = 0.3

See w_min_dtod.

w_min: float = -0.6

Mean of hard bounds across device cross-point ij. The parameters w_min and w_max are used to set the min/max bounds independently.

Note

For this abstract device, we assume that weights can have positive and negative values and are symmetrically around zero. In physical circuit terms, this might be implemented as a difference of two resistive elements.

w_min_dtod: float = 0.3

Device-to-device variation of the hard bounds, of min and max value, respectively. All are given in relative units to w_min, or w_max, respectively.

class aihwkit.simulator.configs.devices.SoftBoundsDevice(construction_seed=0, corrupt_devices_prob=0.0, corrupt_devices_range=1000, diffusion=0.0, diffusion_dtod=0.0, dw_min=0.001, dw_min_dtod=0.3, dw_min_std=0.3, enforce_consistency=True, lifetime=0.0, lifetime_dtod=0.0, perfect_bias=False, reset=0.01, reset_dtod=0.0, reset_std=0.01, up_down=0.0, up_down_dtod=0.01, w_max=0.6, w_max_dtod=0.3, w_min=- 0.6, w_min_dtod=0.3, mult_noise=True)

Bases: aihwkit.simulator.configs.devices.PulsedDevice

Pulsed update behavioral model: soft bounds.

Pulsed update behavioral model, where the update step response size of the material is linearly dependent and it goes to zero at the bound.

This model is based on LinearStepResistiveDevice with parameters set to model soft bounds.

mult_noise: bool = True

Whether to use multiplicative noise instead of additive cycle-to-cycle noise

class aihwkit.simulator.configs.devices.TransferCompoundDevice(unit_cell_devices=<factory>, gamma=0.0, gamma_vec=<factory>, transfer_every=0.0, no_self_transfer=True, transfer_every_vec=<factory>, units_in_mbatch=True, n_cols_per_transfer=1, with_reset_prob=0.0, random_column=False, transfer_lr=1.0, transfer_lr_vec=<factory>, scale_transfer_lr=True, transfer_forward=<factory>, transfer_update=<factory>)

Bases: aihwkit.simulator.configs.devices.UnitCellDevice

Abstract device model that takes 2 or more devices and implements a ‘transfer’ based learning rule.

It uses a (partly) hidden weight (where the SGD update is accumulated), which then is transferred partly and occasionally to the visible weight.

The rate of transfer (e.g. learning rate and how often and how many columns per transfer) and the type (ie. with ADC or without, with noise etc.) can be adjusted.

The weight that is seen in the forward and backward pass is governed by the \(\gamma\) weightening setting.

In principle, a deeper chain of transferred weights can be setup, however, only the device parameters of the first versus the others can be different. However, all devices need to be specified in the list.

Note

Here the devices could be either transferred in analog (essentially within the unit cell) or on separate arrays (using the usual (non-ideal) forward pass and update steps. This can be set with transfer_forward and transfer_update.

as_bindings()

Return a representation of this instance as a simulator bindings object.

Return type

aihwkit.simulator.rpu_base.devices.TransferResistiveDeviceParameter

gamma: float = 0.0

Weightening factor g**(n-1) W[0] + g**(n-2) W[1] + .. + g**0 W[n-1]

gamma_vec: List[float]

User-defined weightening can be given as a list if weights in which case the default weightening scheme with gamma is not used.

n_cols_per_transfer: int = 1

How many consecutive columns to read (from one tile) and write (to the next tile) every transfer event. For read, the input is a 1-hot vector. Once the final column is reached, reading starts again from the first.

no_self_transfer: bool = True

Whether to set the transfer rate of the last device (which is applied to itself) to zero.

random_column: bool = False

Whether to select a random starting column for each transfer event and not take the next column that was previously not transferred as a starting column (the default).

scale_transfer_lr: bool = True

Whether to give the transfer_lr in relative units, ie whether to scale the transfer LR with the current LR of the SGD.

transfer_every: float = 0.0

Transfers every \(n\) mat-vec operations (rounded to multiples/ratios of m_batch for CUDA). If units_in_mbatch is set, then the units are in m_batch instead of mat-vecs, which is equal to the overall the weight re-use during a while mini-batch.

If 0 it is set to x_size / n_cols_per_transfer.

The higher transfer cycles are geometrically scaled, the first is set to transfer_every. Each next transfer cycle is multiplied by by x_size / n_cols_per_transfer.

transfer_every_vec: List[float]

A list of \(n\) entries, to explicitly set the transfer cycles lengths. In this case, the above defaults are ignored.

transfer_forward: IOParameters

Input-output parameters AnalogTileInputOutputParameters that define the read (forward) of an transfer event. For instance the amount of noise or whether transfer is done using a ADC/DAC etc.

transfer_lr: float = 1.0

Learning rate (LR) for the update step of the transfer event. Per default all learning rates are identical. If scale_transfer_lr is set, the transfer LR is scaled by current learning rate of the SGD.

Note

LR is always a positive number, sign will be correctly applied internally.

transfer_lr_vec: List[float]

Transfer LR for each individual transfer in the device chain can be given.

transfer_update: UpdateParameters

Update parameters AnalogTileUpdateParameters that define the type of update used for each transfer event.

units_in_mbatch: bool = True

If set, then the cycle length units of transfer_every are in m_batch instead of mat-vecs, which is equal to the overall the weight re-use during a while mini-batch.

with_reset_prob: float = 0.0

Whether to apply reset of the columns that were transferred with a given probability.

class aihwkit.simulator.configs.devices.UnitCellDevice(unit_cell_devices=<factory>)

Bases: object

Parameters that modify the behaviour of a unit cell.

as_bindings()

Return a representation of this instance as a simulator bindings object.

Return type

aihwkit.simulator.rpu_base.devices.VectorResistiveDeviceParameter

unit_cell_devices: List

Devices that compose this unit cell.

class aihwkit.simulator.configs.devices.VectorUnitCellDevice(unit_cell_devices=<factory>, single_device_update=False, single_device_update_random=False)

Bases: aihwkit.simulator.configs.devices.UnitCellDevice

Abstract resistive device that combines multiple pulsed resistive devices in a single ‘unit cell’.

For instance, a vector device can consist of 2 resistive devices where the sum of the two resistive values are coded for each weight of a cross point.

as_bindings()

Return a representation of this instance as a simulator bindings object.

Return type

aihwkit.simulator.rpu_base.devices.VectorResistiveDeviceParameter

single_device_update: bool = False

Whether to only cycle one device during pulsed update or pulse all devices of one crosspoint at once.

single_device_update_random: bool = False

Whether to select at random (in case of single_device_update)