aihwkit.simulator.parameters.inference module
Inference related parameters for resistive processing units.
- class aihwkit.simulator.parameters.inference.DriftParameter(nu=0.0, t_0=1.0, reset_tol=1e-07, nu_dtod=0.0, nu_std=0.0, wg_ratio=1.0, g_offset=0.0, w_offset=0.0, nu_k=0.0, log_g0=0.0, w_noise_std=0.0)[source]
Bases:
SimpleDriftParameter
Parameter for a power law drift.
The drift is based on the model described by Oh et al (2019).
It computes: .. math:
w_{ij}*\left(\frac{t + \Delta t}{t_0}\right)^(-\nu^\text{actual}_{ij})
where the drift coefficient is drawn once at the beginning and might depend on device. It also can depend on the actual weight value.
The actual drift coefficient is computed as: .. math:
\nu_{ij}^\text{actual} = \nu_{ij} - \nu_k \log \frac{(w_{ij} - w_\text{off}) / r_\text{wg} + g_\text{off}}{G_0} + \nu\sigma_\nu\xi
here \(w_{ij}\) is the actual weight and nu_{ij} fixed for each device given by the mean \(\nu\) and the device-to-device variation: \(\nu_{ij} = \nu + \nu_dtod\nu\xi\) and are only drawn once at the beginning (tile instantiation). xi is Gaussian noise.
Note
If the weight has changed from the last drift call (determined by the
reset_tol
parameter), for instance due to update, decay or noise, then the drift time \(t\) will be reset and start from new, however, the drift coefficients \(\nu_{ij}\) are not changed. On the other hand, if the weights has not changed since last call, \(t\) will accumulate the time.Caution
Note that the drift coefficient does not depend on the initially programmed weight value at \(t=0\) in the current implementation (ie G0 is a constant for all devices), but instead on the actual weight. In some materials (e.g. phase changed materials), that might be not accurate.
- Parameters:
nu (float) –
t_0 (float) –
reset_tol (float) –
nu_dtod (float) –
nu_std (float) –
wg_ratio (float) –
g_offset (float) –
w_offset (float) –
nu_k (float) –
log_g0 (float) –
w_noise_std (float) –
- g_offset: float = 0.0
g_min
to convert to physical units.
- log_g0: float = 0.0
Log g0.
- nu_dtod: float = 0.0
Device-to-device variation of the \(\nu\) values.
- nu_k: float = 0.0
nu with \(W\).
That is \(\nu(R) = nu_0 - k \log(G/G_0)\). See Oh et al. for details.
- Type:
Variation of math
- nu_std: float = 0.0
Cycle-to-cycle variation of \(\nu\).
A more realistic way to add noise of the drift might be using
w_noise_std
.
- w_noise_std: float = 0.0
Additional weight noise (Gaussian diffusion) added to the weights after the drift is applied.
- w_offset: float = 0.0
w(g_min)
, i.e. to what valueg_min
is mapped to in w-space.
- wg_ratio: float = 1.0
(w_max-w_min)/(g_max-g_min)
to convert to physical units.
- class aihwkit.simulator.parameters.inference.SimpleDriftParameter(nu=0.0, t_0=1.0, reset_tol=1e-07)[source]
Bases:
_PrintableMixin
Parameter for a simple power law drift.
The drift as a simple power law drift without device-to-device variation or conductance dependence.
It computes: .. math:
w_{ij}*\left(\frac{t + \Delta t}{t_0}\right)^(-\nu)
- Parameters:
nu (float) –
t_0 (float) –
reset_tol (float) –
- bindings_module: ClassVar[str] = 'devices'
- nu: float = 0.0
Average drift \(\nu\) value.
Need to non-zero to actually use the drift.
- reset_tol: float = 1e-07
Reset tolerance.
This should a number smaller than the expected weight change as it is used to detect any changes in the weight from the last drift call. Every change to the weight above this tolerance will reset the drift time.
Caution
Any write noise or diffusion on the weight might thus interfere with the drift.
- t_0: float = 1.0
Time between write and first read.
Usually assumed in milliseconds, however, it really determines the time units of
time_since_last_call
when calling the drift.
- class aihwkit.simulator.parameters.inference.WeightClipParameter(fixed_value=-1.0, sigma=2.5, type=<factory>)[source]
Bases:
_PrintableMixin
Parameter that clip the weights during hardware-aware training.
Important
A clipping
type
has to be set before any of the parameter changes take any effect.- Parameters:
fixed_value (float) –
sigma (float) –
type (WeightClipType) –
- bindings_class
alias of
WeightClipParameter
- fixed_value: float = -1.0
Clipping value in case of
FixedValue
type.Caution
If
fixed_value > 0
it will be also applied during other clipping types.
- sigma: float = 2.5
Sigma value for clipping for the
LayerGaussian
type.
- type: WeightClipType
Type of clipping.
- class aihwkit.simulator.parameters.inference.WeightModifierParameter(std_dev=0.0, per_batch_sample=False, res=0.0, sto_round=False, dorefa_clip=0.6, pdrop=0.0, enable_during_test=False, rel_to_actual_wmax=True, assumed_wmax=1.0, copy_last_column=False, coeffs=<factory>, type=<factory>, g_max=25.0, pcm_zero_thres=0.0, pcm_t_inference=0.0, pcm_prob_at_reset=0.0, pcm_prob_at_gmax=0.0, pcm_prob_at_random=0.0, pcm_t0=20.0)[source]
Bases:
_PrintableMixin
Parameter that modify the forward/backward weights during hardware-aware training.
- Parameters:
std_dev (float) –
per_batch_sample (bool) –
res (float) –
sto_round (bool) –
dorefa_clip (float) –
pdrop (float) –
enable_during_test (bool) –
rel_to_actual_wmax (bool) –
assumed_wmax (float) –
copy_last_column (bool) –
coeffs (List[float]) –
type (WeightModifierType) –
g_max (float) –
pcm_zero_thres (float) –
pcm_t_inference (float) –
pcm_prob_at_reset (float) –
pcm_prob_at_gmax (float) –
pcm_prob_at_random (float) –
pcm_t0 (float) –
- assumed_wmax: float = 1.0
Assumed weight value that is mapped to the maximal conductance.
This is typically 1.0. This parameter will be ignored if
rel_to_actual_wmax
is set.
- bindings_module: ClassVar[str] = 'tiles'
- coeffs: List[float]
Coefficients for the
POLY
weight modifier type.See
WeightModifierType
for details.
- copy_last_column: bool = False
Whether to not apply noise to the last column (which usually contains the bias values).
- dorefa_clip: float = 0.6
Parameter for DoReFa.
- enable_during_test: bool = False
Whether to use the last modified weight matrix during testing.
Caution
This will not remove drop connect or any other noise during evaluation, and thus should only used with care.
- g_max: float = 25.0
PCM_NOISE and PROG_NOISE parameter, \(g_\text{max}\) setting in \(\mu S\).
- pcm_prob_at_gmax: float = 0.0
PCM_NOISE parameter, probability of devices being at \(g_\text{max}\).
- pcm_prob_at_random: float = 0.0
PCM_NOISE parameter, probability of devices being at random value in the range.
- pcm_prob_at_reset: float = 0.0
PCM_NOISE parameter, probability of reset.
- pcm_t0: float = 20.0
PCM_NOISE parameter, programming conversion time in seconds.
- pcm_t_inference: float = 0.0
PCM_NOISE parameter, time of inference.
- pcm_zero_thres: float = 0.0
PCM_NOISE parameter
- pdrop: float = 0.0
Drop connect probability.
Drop connect sets weights to zero with the given probability. This implements drop connect.
Important
Drop connect can be used with any other modifier type in combination.
- per_batch_sample: bool = False
Should we resample noise for each sample in the batch.
This parameter only affects is used when using the
TorchSimulatorTile
. In case ofRPUCudaTile
it will throw an error.
- rel_to_actual_wmax: bool = True
Whether to calculate the abs max of the weight and apply noise relative to this number.
If set to False,
assumed_wmax
is taken as relative units.
- res: float = 0.0
Resolution of the discretization.
The invert of
res
gives the number of equal sized steps in \(-a_\text{max}\ldots,a_\text{max}\) where the \(a_\text{max}\) is either given by the abs max (ifrel_to_actual_wmax
is set) orassumed_wmax
otherwise.res
is only used in the modifier typesDoReFa
,Discretize
, andDiscretizeAddNormal
.
- std_dev: float = 0.0
Standard deviation of the added noise to the weight matrix.
This parameter affects the modifier types
AddNormal
,MultNormal
andDiscretizeAddNormal
.Note
If the parameter
rel_to_actual_wmax
is set then thestd_dev
is computed in relative terms to the abs max of the given weight matrix, otherwise it in relative terms to the assumed max, which is set byassumed_wmax
.
- sto_round: bool = False
Whether the discretization is done with stochastic rounding enabled.
sto_round
is only used in the modifier typesDoReFa
,Discretize
, andDiscretizeAddNormal
.
- type: WeightModifierType
Type of the weight modification.
- class aihwkit.simulator.parameters.inference.WeightRemapParameter(remapped_wmax=1.0, max_scale_range=0.0, max_scale_ref=0.0, type=<factory>)[source]
Bases:
_PrintableMixin
Parameter that remap the weights during hardware-aware training.
Important
A remap
type
has to be set before any of the parameter changes take any effect.- Parameters:
remapped_wmax (float) –
max_scale_range (float) –
max_scale_ref (float) –
type (WeightRemapType) –
- bindings_class
alias of
WeightRemapParameter
- max_scale_range: float = 0.0
Maximal range of scale values. Use zero to turn any restrictions off (default).
- max_scale_ref: float = 0.0
Reference scale that use used as minimal scale for determining the scale range.
- remapped_wmax: float = 1.0
Assumed max of weight, ie the value of the weight the maximal conductance is mapped to. Typically 1.0.
- type: WeightRemapType
Type of clipping.