Metadata-Version: 2.1
Name: limoe
Version: 0.0.4
Summary: LiMoE - Pytorch
Home-page: https://github.com/kyegomez/LIMoE
License: MIT
Keywords: artificial intelligence,deep learning,optimizers,Prompt Engineering
Author: Kye Gomez
Author-email: kye@apac.ai
Requires-Python: >=3.6,<4.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.9
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Dist: einops
Requires-Dist: swarms-torch
Requires-Dist: torch
Requires-Dist: torchvision
Requires-Dist: zetascale
Project-URL: Documentation, https://github.com/kyegomez/LIMoE
Project-URL: Repository, https://github.com/kyegomez/LIMoE
Description-Content-Type: text/markdown

[![Multi-Modality](agorabanner.png)](https://discord.gg/qUtxnK2NMf)

# LiMoE
Implementation of the "the first large-scale multimodal mixture of experts models." from the paper: "Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts". [CLICK HERE FOR THE PAPER LINK:](https://arxiv.org/abs/2206.02770)


## install
`pip install limoe`

## usage
```python

import torch
from limoe.main import LiMoE

# Text tokens (batch, sequence length)
text = torch.randint(0, 100, (1, 64))

# image (batch, channels, height, width)
image = torch.randn(1, 3, 224, 224)

# Create an instance of LiMoE with the specified parameters
model = LiMoE(
    dim=64,  # Dimension of the input and output tensors
    depth=3,  # Number of layers in the encoder
    heads=8,  # Number of attention heads
    num_tokens=100,  # Number of tokens in the vocabulary
    seq_length=64,  # Length of the input sequence
    num_experts=4,  # Number of experts in the mixture-of-experts layer
    dim_head=64,  # Dimension of each attention head
    dropout=0.1,  # Dropout rate
    ff_mult=4,  # Multiplier for the dimension of the feed-forward layer
    patch_size=16,  # Patch size
    image_size=224,  # Image size
    channels=3,  # Number of image channels
    dense_encoder_depth=5
)

# Pass the input tensor through the model and print the output
out = model(text, image)

# Print
print(out)
```

# License
MIT

