Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[LayerNorm] Support inconsecutive dimensions as axes #69

Open
lisa0314 opened this issue Jan 7, 2025 · 0 comments
Open

[LayerNorm] Support inconsecutive dimensions as axes #69

lisa0314 opened this issue Jan 7, 2025 · 0 comments
Labels

Comments

@lisa0314
Copy link
Collaborator

lisa0314 commented Jan 7, 2025

Using reduceMean and add, mul, div, sqrt to emulation when axes are inconsecutive.

function layerNormalization(builder, input, options) {
// The reduction of the mean and variance values happens over the spatial
// dimensions across all the input features (i.e. all channels) of the input
// tensor.
const reduceOptions = {axes: [1, 2, 3], keepDimensions: true};
const mean = builder.reduceMean(input, reduceOptions);
const variance = builder.reduceMean(
builder.pow(builder.sub(input, mean), builder.constant(input.dataType, 2)),
reduceOptions);

// The scale and bias tensors are of the shape of the input
// specified by the values in the axes parameter (i.e. [1,2,3]).
return builder.add(
builder.mul(
options.scale,
builder.div(
builder.sub(input, mean),
builder.sqrt(builder.add(variance, options.epsilon)))),
options.bias);
}

@lisa0314 lisa0314 changed the title ONNX layer norm only supports last consecutive dimensions as axes Support inconsecutive dimensions as axes for layernormalization Jan 8, 2025
@lisa0314 lisa0314 changed the title Support inconsecutive dimensions as axes for layernormalization [LayerNorm] Support inconsecutive dimensions as axes Jan 13, 2025
@huningxin huningxin added the ort label Feb 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants