Module mixtral 

Mixtral Model, a sparse mixture of expert model based on the Mistral architecture

See Mixtral model details at:

• Hugging Face
• Mixtral-8x7B Blog Post

The model uses a mixture of experts architecture with:

• 8 experts per layer
• Top 2 expert routing
• Sliding window attention
• RoPE embeddings

References:

• Hugging Face Implementation
• Mixtral Blog Post

Structs

Config

https://github.com/huggingface/transformers/blob/1a585c1222a56bcaecc070966d558d4a9d862e83/src/transformers/models/mixtral/configuration_mixtral.py#L113

Model