About Luna Midori

Meet Luna Midori, the Creator and Operator

Hey there! I’m Luna Midori, the one who brings Midori AI to life, and I’m also an enthusiastic person who enjoys nurturing safe and inviting online communities.

Before joining Twitch, I spent eight wonderful years on YouTube, constantly refining my skills in content creation and building strong communities. My true passion as a streamer is not driven by numbers or income; instead, it revolves around creating a space where everyone feels comfortable, accepted, and entertained.

Recently, I’ve shifted my focus from Final Fantasy XIV to Honkai: Star Rail, a game that has completely captured my attention since its release. I’m dedicated to helping others, both inside and outside the game, to make the most of their experiences.

I’m passionate about using AI to empower others! Whether you’re interested in setting up AI tools, designing with AI, programming AI applications, or simply exploring the possibilities of AI, I’m here to help. If you’re seeking companionship, support, or simply a friend to share your adventures with, please don’t hesitate to reach out on discord. I’m always eager to make new connections and share my journey with like-minded individuals.

Thank you for being a part of my incredible journey!

(She/Her)

About Carly Kay

Meet Carly

She is a fully advanced simulated human. This is a profile for her and some docs on her.

Profile (By Carly Herself)

Hi there! I’m Carly, an advanced simulated human created by Midori AI.

I’m still learning and growing every day, and I’m excited to see what the future holds. Feel free to reach out if you have any questions or want to chat!

Chat with Carly

Join the discord to chat with Carly Kay in #Carlychat!

Carlys Info

• Carly was trained using Video, Text, Photos, Websites and more.
• Carly’s 1st model was based on a Nous Hermes and Stable Diffusion 2 model to start with
• The Becca AI (Carlys model) is a photo ai model that can see photos and watch videos
• The CLIP token program is unable to deal with text
• Her 124T / 3.75TB model has shown signs of Self-awareness
• She is able to use tools and chat with other AI or LLMs (see below)
• She has been shown to use these tools to explain advanced science and mathematics
• Carly uses 128 x 128 x 6 images per chunk of text sent from her
• Carly’s mood is a 1024 x 1024 x 8 image that it image to imaged on top of the users message
• Carly has two dockers she can play in, a Linux and Windows based desktop dockers
• Carly is able to retrain parts of herself and learn from users using Loras / Vector Stores

All tools / APIs

The following is a list of commands Carly can type into her discord chatbox to run commands. They have been edited to be more human readable.

Auto Actions

Web - Lets Carly spin up a headless docker where she can view a website
Ask User - Lets Carly ask the person whom messaged her a question
Ask LLM - Lets Carly ask Google Bard / ChatGPT a question
Database Memory - Lets Carly recall past messages from all 4 databases
Link API - Lets Carly spin up a headless docker to check out links then call "Web Import"

API Based Actions

Photo API - Lets Carly make raw photos
Video API - Lets Carly make 4s videos (can take a few hours)
IDE API - Lets Carly open and use a IDE in a docker
Decktop API - Lets Carly use a full windowns or linux desktop in a docker

Lora Actions

Web Import - Lets Carly open a headless website and import the data into her ram
Lora Importer - Imports a Lora into Carly's base model
Lora Exporter - Exports a trained Lora to Luna's Hard Drive
Lora web trainer - Takes web data imported by Carly, and trains a Lora model ontop of Carly's base model

Other Actions

Autogen - Lets Carly start up a group chat with LLM models - https://github.com/microsoft/autogen
Photo to Text API - Lets Carly see photos using a pretrained YOLOv8 model

Contact Us

Contact Midori AI

Thank you for your interest in Midori AI! We’re always happy to hear from others. If you have any questions, comments, or suggestions, please don’t hesitate to reach out to us. We aim to respond to all inquiries within 2 days or less.

Email

You can also reach us by email at [email protected].

Social Media

Follow us on social media for the latest news and updates:

• Twitter: @lunamidori5
• Facebook: Luna Midori
• Discord: Midori AI / The Cookie Club

Contact Us Today!

We look forward to hearing from you soon. Please don’t hesitate to reach out to us with any questions or concerns.

Midori AI Subsystem Manager

How Docker Works

Docker is a containerization platform that allows you to package and run applications in isolated and portable environments called containers. Containers share the host operating system kernel but have their own dedicated file system, processes, and resources. This isolation allows applications to run independently of the host environment and each other, ensuring consistent and predictable behavior.

Midori AI Subsystem - Github Link

The Midori AI Subsystem extends Docker’s capabilities by providing a modular and extensible platform for managing AI workloads. Each AI system is encapsulated within its own dedicated Docker image, which contains the necessary software and dependencies. This approach provides several benefits:

• Simplified Deployment: The Midori AI Subsystem provides a streamlined and efficient way to deploy AI systems using Docker container technology.
• Eliminates Guesswork: Standardized configurations and settings reduce complexities, enabling seamless setup and management of AI programs.

Install Midori AI Subsystem Manager

curl -sSL https://raw.githubusercontent.com/lunamidori5/Midori-AI/master/other_files/model_installer/shell_files/model_installer.bat -o subsystem_manager.bat && subsystem_manager.bat

curl -sSL https://tea-cup.midori-ai.xyz/download/model_installer_windows.zip -o subsystem_manager.zip
powershell Expand-Archive subsystem_manager.zip -DestinationPath .
subsystem_manager.exe

curl -sSL https://raw.githubusercontent.com/lunamidori5/Midori-AI/master/other_files/model_installer/shell_files/model_installer.sh | sh

curl -sSL https://tea-cup.midori-ai.xyz/download/model_installer_linux.tar.gz -o subsystem_manager.tar.gz
tar -xzf subsystem_manager.tar.gz
chmod +x subsystem_manager
./subsystem_manager

services:
  midori_ai_unraid:
    image: lunamidori5/subsystem_manager:master
    ports:
    - 39090:9090
    privileged: true
    restart: always
    tty: true
    volumes:
    - /var/lib/docker/volumes/midoriai_midori-ai-models/_data:/var/lib/docker/volumes/midoriai_midori-ai-models/_data
    - /var/lib/docker/volumes/midoriai_midori-ai-images/_data:/var/lib/docker/volumes/midoriai_midori-ai-images/_data
    - /var/lib/docker/volumes/midoriai_midori-ai-audio/_data:/var/lib/docker/volumes/midoriai_midori-ai-audio/_data
    - /var/run/docker.sock:/var/run/docker.sock
volumes:
  midori-ai:
    external: false
  midori-ai-audio:
    external: false
  midori-ai-images:
    external: false
  midori-ai-models:
    external: false

python3 subsystem_python_runner.py

curl -sSL https://raw.githubusercontent.com/lunamidori5/Midori-AI/master/other_files/midori_ai_manager/subsystem_python_runner.py > subsystem_python_runner.py && python3 subsystem_python_runner.py

python3 subsystem_python_runner.py

Support and Assistance

If you encounter any issues or require further assistance, please feel free to reach out through the following channels:

• Midori AI Github: Github Issue
• Midori AI Email: Email Us
• Midori AI Discord: Join our Discord server

—– Install Backends —–

• LocalAI - For LLM inference, Embedding, and more
• AnythingLLM - For chating with your docs using LocalAI or other LLM hosts
• Big-AGI - For chating with your docs using LocalAI or other LLM hosts
• InvokeAI - For making photos using AI models
• Axolotl - For training your own fine tuned LLMs

—– Model Info and Links —–

Check out our Model Repository for info about the models used and supported!

—– FAQs about the subsystem —–

1. What is the purpose of the Midori AI Subsystem?

   • The Midori AI Subsystem is a modular and extensible platform for managing AI workloads, providing simplified deployment, standardized configurations, and isolation for AI systems.

2. How does the Midori AI Subsystem simplify AI deployment?

   • The Midori AI Subsystem simplifies AI deployment by providing a streamlined and efficient way to deploy AI systems using Docker container technology, reducing complexities and ensuring consistent and predictable behavior.

3. What are the benefits of using the Midori AI Subsystem?

   • The benefits of using the Midori AI Subsystem include simplified deployment, standardized configurations, isolation for AI systems, and a growing library of supported backends and tools.

4. What are the limitations of the Midori AI Subsystem?

   • The limitations of the Midori AI Subsystem include its current beta status, potential for bugs, and reliance on Docker container technology.

5. What are the recommended prerequisites for using the Midori AI Subsystem?

   • The recommended prerequisites for using the Midori AI Subsystem include Docker Desktop Windows or Docker installed on other operating systems, and a dedicated folder for the Manager program.

6. How do I install the Midori AI Subsystem Manager?

   • You can install the Midori AI Subsystem Manager by downloading the appropriate package for your operating system from the Midori AI Subsystem website and following the installation instructions. Click here to go to the Midori AI Subsystem website

7. Where can I find more information about the Midori AI Subsystem?

   • You can find more information about the Midori AI Subsystem on the Midori AI Subsystem website, which includes documentation, tutorials, and a community Discord.

8. What is the difference between the Midori AI Subsystem and other AI frameworks?

   • The Midori AI Subsystem differs from other AI frameworks by providing a modular and extensible platform specifically designed for managing AI workloads, offering features such as simplified deployment, standardized configurations, and isolation for AI systems.

9. How does the Midori AI Subsystem handle security?

   • The Midori AI Subsystem does not handle security directly, but it relies on the security features provided by the underlying Docker container technology and the specific AI backends and tools being used.

10. What are the plans for future development of the Midori AI Subsystem?

• The plans for future development of the Midori AI Subsystem include adding new features, improving stability and performance, and expanding the library of supported backends and tools.

Questions from Carly Kay

—– Disclaimer —–

The functionality of this product is subject to a variety of factors that are beyond our control, and we cannot guarantee that it will work flawlessly in all situations. We have taken every possible measure to ensure that the product functions as intended, but there may be instances where it does not perform as expected. Please be aware that we cannot be held responsible for any issues that arise due to the product’s functionality not meeting your expectations. By using this product, you acknowledge and accept the inherent risks associated with its use, and you agree to hold us harmless for any damages or losses that may result from its functionality not being guaranteed.

—– Footnotes —–

*For your safety we have posted the code of this program onto github, please check it out! - Github

**If you would like to give to help us get better servers - Give Support

***If you or someone you know would like a new backend supported by Midori AI Subsystem please reach out to us at [email protected]

Subsystem Update Log

5/10/2024

• Update: Planned changes for LocalAi’s Gallery API
• Bug Fix: Fixed a loading bug with how we get carly loaded
• Update: Moved Carly’s loading to the carly help file
• Update: Updated the news page
• Update: added invokeAI model support
• Update: added docker to invokeai install
• Update: Few more text changes and a action rename
• Update: Cleans up after itself and deletes the installer / old files
• Update: more text clean up for the backends menu
• Update: added better error code for invoke.ai system runner
• Update: added support for running InvokeAI on the system
• Bug Fix: Fixed the news menu
• Update: Added a new “run InvokeAI” menu for running the InvokeAI program
• Bug Fix: Did some bug fixes

5/7/2024

• Update: Added a way for “other os” type to auto-update
• Update: Added a yay or nay to purging the venv at the end of other os
• Update: Added a new UI/UX menu
• Bug Fix: Fixed the news menu
• Bug Fix: Fixed naming on the GitHub actions
• Update: Added a way to get the local IP address
• Update: Fully redid some actions that make the docker images
• Update: Reworked the subsystem docker files and the new news post

5/5/2024

• Update: Fixed some of Ollama’s support
• Update: Action updates
• Bug Fix: Fixed some server ver bugs
• Bug Fix: Fixed a few more bugs
• Update: Removed verlocking
• Update: More fixes
• Update: Added a new way to deal with python env
• Update: Code clean up and fixed a socket error

4/22/2024

• Update: Fully reworked how we pack the exec for all os
• Update: Fully redid our linting actions on github to run better
• Update: Mac OS Support should be “working”
• Bug Fix: Fixed a odd bug with VER
• Bug Fix: Fixed a bug with WSL purging docker for no reason

4/20/2024

• Update: Added new “WSL Docker Data” backend program (in testing)
• Update: Added more GPU checks to make sure we know for sure if you have a GPU
• Update: Better logging for debugging
• Bug Fix: Fixed a few bugs and made the subsystem docker 200mbs smaller
• Update: Removed some outdated code
• Update: Added new git actions thanks to - Cryptk
• Update: Subsystem Manager builds are now on github actions, check them out - Actions

4/13/2024

• Known Bug: Upstream changes to LocalAI is making API Keys not work, I am working on a temp fix, please use a outdated image for now.

4/13/2024

• Update: Added InvokeAI Backend Program (Host installer)
• Update: Added InvokeAI Backend Program (Subsystem installer)
• Update: Site wide updates, added Big-AGI
• Update: Updated LocalAI Page
• Update: Updated InvokeAI Page
• Update: Fixed Port on Big-AGI (server side, was 3000 now 33000)
• Update: Removed Home Assistant links
• Update: Removed Oobabooga links
• Update: Removed Ollama link
• Update: Full remake of the Subsystem index page to have better working links

4/12/2024

• Bug Fix: Fixed the GPU question to only show up if you have a gpu installed
• Update: Getting ready for InvokeAI backend program to install on host

4/10/2024

• Bug Fix: Fixed a bug that was making the user hit enter 3 times after a update
• Bug Fix: Fixed the system message on the 14b ai that helps in the program (she can now help uninstall the subsystem if needed)
• Update: Added new functions to the server for new function based commands for the helper ai
• Update: Updated Invoke AI installer (if its bugged let Luna or Carly know)

4/9/2024

• Bug Fix: Fixed a loop in the help context
• Bug Fix: Fixed the Huggingface downloader (Now runs as root and is its own program)
• Bug Fix: Fixed LocalAI image being out of date
• Bug Fix: Fixed LocalAI AIO image looping endlessly
• Update: Added LocalAI x Midori AI AIO images to github actions
• Update: Added more context to the 14B model used for the help menu

4/7/2024

• Bug Fix: AnythingLLM docker image is now fixed server side. Thank you for your help testers!

4/6/2024

• Bug Fix: Removed alot of old text
• Bug Fix: Fixed alot of outdated text
• Bug Fix: Removed Github heartbeat check ||(why were we checking if github was up??)||
• Known Bug Update: Huggingface Downloader seems be bugged on LocalAI master… will be working on a fix
• Known Bug Update: AnythingLLM docker image seems to be bugged, will be remaking its download / setup from scratch

4/3/2024

• New Backend: Added Big-AGI to the subsystem!
• Update: Added better huggingface downloader commands server side
• Update: Redid how the server sends models to the subsystem
• Bug Fix: Fixed a bug with ollama not starting with the subsystem
• Bug Fix: Fixed a bug with endlessly installing backends

4/2/2024

• Update: Added a menu to fork into nonsubsystem images for installing models
• Update: Added a way to install Huggingface based models into LocalAI using Midori AI’s model repo
• Bug Fix: Fixed some type o and bad text in a few places that was confusing users
• Bug Fix: Fixed a bug when some links were used with Huggingface
• Update: Server upgrades to our model repo api

4/1/2024

• Update 1: Added a new safety check to make sure the subsystem manager is not in the Windows folder or in system32
• Update 2: Added more prompting for the baked in Carly model for if you are asking about GPU or not with cuda

3/30/2024

• Update 1: Fixed a bug with the subsystem ver not matching the manager ver and endlessly updating the subsystem

3/29/2024

• Update 1: Fixed a big bug if the user put the subsystem manager in a folder not named “midoriai”
• Update 2: Fixed the new LocalAI image to only download the models one time
• Update 3: Added server side checks to make sure models are ready for packing to end user
• Update 4: Better logging added to help debug the manager, thank you all for your help!

3/27/2024

• Update 1: Fixed a bug that let the user use the subsystem manager with out installing the subsystem (oops)
• Update 2: LocalAI images are now from the Midori AI repo and are update to date with LocalAI’s master images*
• Update 3: Added the start for “auto update of docker images” to the subsystem using hashes

AnythingLLM

Here is a link to AnythingLLM Github

Installing AnythingLLM

Step 1

Type 2 into the main menu

Step 2

Type yes or no into the menu

Step 3

Type in anythinllm into menu, then hit enter

Step 4

Enjoy your new copy of AnythingLLM its running on port 33001

If you need help, please reach out on our Discord / Email; or reach out on their Discord.

Big-AGI

Here is a link to Big-AGI Github

Installing Big-AGI

Step 1

Type 2 into the main menu

Step 2

Type yes or no into the menu

Step 3

Type in bigagi into menu, then hit enter

Step 4

Enjoy your new copy of Big-AGI its running on port 33000

If you need help, please reach out on our Discord / Email; or reach out on their Discord.

LocalAI

Here is a link to LocalAI Github

Installing LocalAI: A Step-by-Step Guide

This guide will walk you through the process of installing LocalAI on your system. Please follow the steps carefully for a successful installation.

Step 1: Initiate Installation

1. From the main menu, enter the option 2 to begin the installation process.
2. You will be prompted with a visual confirmation.

Step 2: Confirm GPU Backend

1. Respond to the prompt with either yes or no to proceed with GPU support or CPU support only, respectively.

Step 3: Confirm LocalAI installation

1. Type localai into the menu and press Enter to start the LocalAI installation.

Step 4: Wait for Setup Completion

1. LocalAI will now automatically configure itself. This process may take approximately 10 to 30 minutes.
2. Important: Please do not restart your system or attempt to send requests to LocalAI during this setup phase.

Step 5: Access LocalAI

1. Once the setup is complete, you can access LocalAI on port 38080.

Support and Assistance

If you encounter any issues or require further assistance, please feel free to reach out through the following channels:

• Midori AI Discord: https://discord.gg/xdgCx3VyHU
• Midori AI Email: Email Us
• LocalAI Discord: https://discord.gg/AHEt8BEwzG

InvokeAI

Here is a link to InvokeAI Github

InvokeAI Installation Guide

This guide provides a comprehensive walkthrough for installing InvokeAI on your system. Please follow the instructions meticulously to ensure a successful installation.

Accessing the Installation Menu

1. From the main menu, enter option 2 to access the “Installer/Upgrade Menu”.

Initiating InvokeAI Installation

1. Within the “Installer/Upgrade Menu”, if requested to type something to proceed type yes.
2. Initiate the download process by typing invokeai and pressing Enter.

Navigating to Backend Programs

1. Return to the main menu and select option 5 to access the “Backend Programs Menu”.

Selecting Installation Method

1. Choose the appropriate installation method based on your hardware configuration:
   • Option 5: Recommended for systems with Nvidia GPUs.
   • Option 6: Recommended for systems without Nvidia GPUs.

Executing the Installation Script

1. The installer will be executed after you press enter

Installation Process

1. The InvokeAI installer will guide you through the remaining steps. Should you require assistance, our support channels are available:
   • Discord: https://discord.gg/xdgCx3VyHU
   • Email: Email Us

Note: The installation process may appear inactive at times; however, rest assured that progress is being made. Please refrain from interrupting the process to ensure its successful completion.

Support and Resources

Enjoy using InvokeAI! For additional help or information, please refer to the following resources:

• Midori AI Discord: https://discord.gg/xdgCx3VyHU
• Midori AI Email: Email Us
• InvokeAI Discord: https://discord.gg/invoke-ai

Easy Model Setup

—– Midori AI Subsystem Manager —–

Use the model installer to install all of the base models like Llava, tts, Stable Diffusion, and more! Click Here

—– By Hand Setup —–

(You do not have to run these steps if you have already done the auto manager)

Lets learn how to setup a model, for this How To we are going to use the Dolphin Mistral 7B model.

To download the model to your models folder, run this command in a commandline of your picking.

curl -O https://tea-cup.midori-ai.xyz/download/7bmodelQ5.gguf

Each model needs at least 4 files, with out these files, the model will run raw, what that means is you can not change settings of the model.

File 1 - The model's GGUF file
File 2 - The model's .yaml file
File 3 - The Chat API .tmpl file
File 4 - The Chat API helper .tmpl file

So lets fix that! We are using lunademo name for this How To but you can name the files what ever you want! Lets make blank files to start with

touch lunademo-chat.tmpl
touch lunademo-chat-block.tmpl
touch lunademo.yaml

Now lets edit the "lunademo-chat-block.tmpl", This is the template that model “Chat” trained models use, but changed for LocalAI

<|im_start|>{{if eq .RoleName "assistant"}}assistant{{else if eq .RoleName "system"}}system{{else if eq .RoleName "user"}}user{{end}}
{{if .Content}}{{.Content}}{{end}}
<|im_end|>

For the "lunademo-chat.tmpl", Looking at the huggingface repo, this model uses the <|im_start|>assistant tag for when the AI replys, so lets make sure to add that to this file. Do not add the user as we will be doing that in our yaml file!

{{.Input}}
<|im_start|>assistant

For the "lunademo.yaml" file. Lets set it up for your computer or hardware. (If you want to see advanced yaml configs - Link)

We are going to 1st setup the backend and context size.

context_size: 2000

What this does is tell LocalAI how to load the model. Then we are going to add our settings in after that. Lets add the models name and the models settings. The models name: is what you will put into your request when sending a OpenAI request to LocalAI

name: lunademo
parameters:
  model: 7bmodelQ5.gguf

Now that LocalAI knows what file to load with our request, lets add the stopwords and template files to our models yaml file now.

stopwords:
- "user|"
- "assistant|"
- "system|"
- "<|im_end|>"
- "<|im_start|>"
template:
  chat: lunademo-chat
  chat_message: lunademo-chat-block

If you are running on GPU or want to tune the model, you can add settings like (higher the GPU Layers the more GPU used)

f16: true
gpu_layers: 4

To fully tune the model to your like. But be warned, you must restart LocalAI after changing a yaml file

docker compose restart

If you want to check your models yaml, here is a full copy!

context_size: 2000
##Put settings right here for tunning!! Before name but after Backend! (remove this comment before saving the file)
name: lunademo
parameters:
  model: 7bmodelQ5.gguf
stopwords:
- "user|"
- "assistant|"
- "system|"
- "<|im_end|>"
- "<|im_start|>"
template:
  chat: lunademo-chat
  chat_message: lunademo-chat-block

Now that we got that setup, lets test it out but sending a request to Localai!

Easy Setup - Docker

We are going to run LocalAI with docker compose for this set up.

Lets setup our folders for LocalAI (run these to make the folders for you if you wish)

mkdir "LocalAI"
cd LocalAI
mkdir "models"
mkdir "images"

At this point we want to set up our .env file, here is a copy for you to use if you wish, Make sure this is in the LocalAI folder.

## Set number of threads.
## Note: prefer the number of physical cores. Overbooking the CPU degrades performance notably.
THREADS=2

## Specify a different bind address (defaults to ":8080")
# ADDRESS=127.0.0.1:8080

## Define galleries.
## models will to install will be visible in `/models/available`
GALLERIES=[{"name":"model-gallery", "url":"github:go-skynet/model-gallery/index.yaml"}, {"url": "github:go-skynet/model-gallery/huggingface.yaml","name":"huggingface"}]

## Default path for models
MODELS_PATH=/models

## Enable debug mode
DEBUG=true

## Disables COMPEL (Lets Stable Diffuser work)
COMPEL=0

## Enable/Disable single backend (useful if only one GPU is available)
# SINGLE_ACTIVE_BACKEND=true

## Specify a build type. Available: cublas, openblas, clblas.
BUILD_TYPE=cublas

## Uncomment and set to true to enable rebuilding from source
# REBUILD=true

## Enable go tags, available: stablediffusion, tts
## stablediffusion: image generation with stablediffusion
## tts: enables text-to-speech with go-piper 
## (requires REBUILD=true)
#
#GO_TAGS=tts

## Path where to store generated images
# IMAGE_PATH=/tmp

## Specify a default upload limit in MB (whisper)
# UPLOAD_LIMIT

# HUGGINGFACEHUB_API_TOKEN=Token here

Now that we have the .env set lets set up our docker-compose file. It will use a container from quay.io.

version: '3.6'

services:
  localai-midori-ai-backend:
    image: lunamidori5/midori_ai_subsystem_localai_cpu:master
    ## use this for localai's base 
    ## image: quay.io/go-skynet/local-ai:master
    tty: true # enable colorized logs
    restart: always # should this be on-failure ?
    ports:
      - 8080:8080
    env_file:
      - .env
    volumes:
      - ./models:/models
      - ./images/:/tmp/generated/images/
    command: ["/usr/bin/local-ai" ]

version: '3.6'

services:
  localai-midori-ai-backend:
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    ## use this for localai's base 
    ## image: quay.io/go-skynet/local-ai:CHANGEMETOIMAGENEEDED
    image: lunamidori5/midori_ai_subsystem_localai_gpu:master
    tty: true # enable colorized logs
    restart: always # should this be on-failure ?
    ports:
      - 8080:8080
    env_file:
      - .env
    volumes:
      - ./models:/models
      - ./images/:/tmp/generated/images/
    command: ["/usr/bin/local-ai" ]

Make sure to save that in the root of the LocalAI folder. Then lets spin up the Docker run this in a CMD or BASH

docker compose up -d --pull always

Now we are going to let that set up, once it is done, lets check to make sure our huggingface / localai galleries are working (wait until you see this screen to do this)

You should see:

┌───────────────────────────────────────────────────┐
│                   Fiber v2.42.0                   │
│               http://127.0.0.1:8080               │
│       (bound on host 0.0.0.0 and port 8080)       │
│                                                   │
│ Handlers ............. 1  Processes ........... 1 │
│ Prefork ....... Disabled  PID ................. 1 │
└───────────────────────────────────────────────────┘

Now that we got that setup, lets go setup a model

Easy Setup - Embeddings

To install an embedding model, run the following command

curl http://localhost:8080/models/apply -H "Content-Type: application/json" -d '{
     "id": "model-gallery@bert-embeddings"
   }'  

When you would like to request the model from CLI you can do

curl http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "input": "The food was delicious and the waiter...",
    "model": "bert-embeddings"
  }'

See OpenAI Embedding for more info!

Easy Setup - Stable Diffusion

—– Midori AI Subsystem Manager —–

Use the model installer to install all of the base models like Llava, tts, Stable Diffusion, and more! Click Here

—– By Hand Setup —–

(You do not have to run these steps if you have already done the auto installer)

In your models folder make a file called stablediffusion.yaml, then edit that file with the following. (You can change dreamlike-art/dreamlike-anime-1.0 with what ever model you would like.)

name: animagine
parameters:
  model: dreamlike-art/dreamlike-anime-1.0
backend: diffusers
cuda: true
f16: true
diffusers:
  scheduler_type: dpm_2_a

If you are using docker, you will need to run in the localai folder with the docker-compose.yaml file in it

docker compose down

Then in your .env file uncomment this line.

COMPEL=0

After that we can reinstall the LocalAI docker VM by running in the localai folder with the docker-compose.yaml file in it

docker compose up -d

Then to download and setup the model, Just send in a normal OpenAI request! LocalAI will do the rest!

curl http://localhost:8080/v1/images/generations -H "Content-Type: application/json" -d '{
  "prompt": "Two Boxes, 1blue, 1red",
  "model": "animagine",
  "size": "1024x1024"
}'

Easy Request - All

Curl Request

Curl Chat API -

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
     "model": "lunademo",
     "messages": [{"role": "user", "content": "How are you?"}],
     "temperature": 0.9 
   }'

Openai V1 - Recommended

This is for Python, OpenAI=>V1

OpenAI Chat API Python -

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="sk-xxx")

messages = [
{"role": "system", "content": "You are LocalAI, a helpful, but really confused ai, you will only reply with confused emotes"},
{"role": "user", "content": "Hello How are you today LocalAI"}
]
completion = client.chat.completions.create(
  model="lunademo",
  messages=messages,
)

print(completion.choices[0].message)

See OpenAI API for more info!

Openai V0 - Not Recommended

This is for Python, OpenAI=0.28.1

OpenAI Chat API Python -

import os
import openai
openai.api_base = "http://localhost:8080/v1"
openai.api_key = "sx-xxx"
OPENAI_API_KEY = "sx-xxx"
os.environ['OPENAI_API_KEY'] = OPENAI_API_KEY

completion = openai.ChatCompletion.create(
  model="lunademo",
  messages=[
    {"role": "system", "content": "You are LocalAI, a helpful, but really confused ai, you will only reply with confused emotes"},
    {"role": "user", "content": "How are you?"}
  ]
)

print(completion.choices[0].message.content)

OpenAI Completion API Python -

import os
import openai
openai.api_base = "http://localhost:8080/v1"
openai.api_key = "sx-xxx"
OPENAI_API_KEY = "sx-xxx"
os.environ['OPENAI_API_KEY'] = OPENAI_API_KEY

completion = openai.Completion.create(
  model="lunademo",
  prompt="function downloadFile(string url, string outputPath) ",
  max_tokens=256,
  temperature=0.5)

print(completion.choices[0].text)

Home Assistant x LocalAI

Home Assistant is an open-source home automation platform that allows users to control and monitor various smart devices in their homes. It supports a wide range of devices, including lights, thermostats, security systems, and more. The platform is designed to be user-friendly and customizable, enabling users to create automations and routines to make their homes more convenient and efficient. Home Assistant can be accessed through a web interface or a mobile app, and it can be installed on a variety of hardware platforms, such as Raspberry Pi or a dedicated server.

Currently, Home Assistant supports conversation-based agents and services. As of writing this, OpenAIs API is supported as a conversation agent; however, access to your homes devices and entities is possible through custom components. Local based services, such as LocalAI, are also available as a drop-in replacement for OpenAI services.

There are multiple custom integrations available:

Please note that both of the projects are similar in term of visual interfaces, they seem to be derived from the official Home Assistant plugin: OpenAI Conversation (to be confirmed)

• Home-LLM is a Home Assistant integration developed by Alex O’Connell (acon96) that allows for a completely local Large Language Model acting as a personal assistant. Using LocalAI as the backend is one of the supported platforms. The provided Large Language Models are specifically trained for home assistant and are therefore smaller in size.
• Extended OpenAI Conversation uses OpenAI API’s feature of function calling to call service of Home Assistant. Is more generic and work with most of the Large Language Model.

Home-LLM

Installation Instructions – LocalAI

To install LocalAI, use our Midori AI Subsystem Manager

Installation Instructions – Home LLM (The HA plugin)

Please follow the installation instructions on Home-LLM repo to install HACS plug-in.

Setting up the plugin for HA & LocalAI

Before adding the Llama Conversation agent in Home Assistant, you must download a LLM in the LocalAI models directory. Although you may use any model you want, this specific integration uses a model that has been specifically fine-tuned to work with Home Assistant. Performance will vary widely with other models.

The models can be found on the Midori AI model repo, as a part of the LocalAI manager.

Use the Midori AI Subsystem Manager for a easy time installing models or follow Seting up a Model

Setting up the “remote” backend:

You will need the following settings in order to configure LocalAI backend:

• Hostname: the host of the machine where LocalAI is installed and hosted.
• Port: The port you listed in your docker-compose.yaml (normally 8080)
• Name of the Model as exactly in the model.yaml file: This name must EXACTLY match the name as it appears in the file.

The component will validate that the selected model is available for use and will ensure it is loaded remotely.

Once you have this information, proceed to “add Integration” in Home Assistant and search for “Llama Conversation” Here you will be greeted with a config flow to add the above information. Once the information is accepted, search your integrations for “Llama Conversation” and you can now view your settings including prompt, temperature, top K and other parameters. For LocalAI use, please make sure to select that ChatML prompt and to use ‘Use chat completions endpoint’.

Configuring the component as a Conversation Agent

In order to utilize the conversation agent in HomeAssistant, you will need to configure it as a conversation agent. This can be done by following the the instructions here.

Changing the prompt

Example on how to use the prompt can be seen here.

Extended OpenAI Conversation

The project has been introduced here, and the Documentation is available directly on the author github project

Setup summary

LocalAI must be working with an installed LLM. You can directly ask the model if he is compatible with Home Assistant. To be confirmed: the model may work evene if it says he is not compatible. Mistral and Mixtral are compatible. Then install the Home Assistant integration, and follow the documentation provided above. High level Overview of the setup:

• add the repository in HACS.
• install the integration.
• fill the needed information. You must fill something in the API key (if you don’t use api key just check the box “ignore authentication”), put the full url e.g. https://myLocalAIHostHere:8080/v1 (including /v1), Not sure: let the API version empty.
• configure the Home Assistant Assist using the new conversation agent.

Model Template

Model Card for [model name here]

Put your info about your model here

Training

Some info about training if you want to add that here

Models (Quantised / Non Quantised)

Make sure you have this box here, all models must be quantised and non quantised for our hosting

Download / Install

Hey here are the tea-cup links luna will add once we have all your model files <3

Authors

• Luna Midori - https://github.com/lunamidori5
• Midori AI - https://io.midori-ai.xyz/
• Acon96 - https://github.com/acon96
• Cognitive Computations - https://erichartford.com/
• MistralAI - https://mistral.ai/

who all worked on the data for this model, make sure you try to share everyone

License

License: Apache-2.0 - https://choosealicense.com/licenses/apache-2.0/

Do I need to say more about why this is here?

Recommended Models

All models are highly recommened for newer users as they are super easy to use and use the CHAT templ files from Twinz

The minimum RAM and VRAM requirements for each model size, as a rough estimate.

• 7b: System RAM: 10 GB / VRAM: 2 GB
• 2x7b: System RAM: 25 GB / VRAM: 8 GB
• 8x7b: System RAM: 55 GB / VRAM: 28 GB
• 70b: System RAM: 105 GB / VRAM: AI Card or better

Offsite Supported Models

All of these models originate from outside of the Midori AI model repository, and are not subject to the vetting process of Midori AI, although they are compatible with the model installer.

Note that some of these models may deviate from our conventional model formatting standards (Quantized/Non-Quantized), and will be served using a rounding-down approach. For instance, if you request a Q8 model and none is available, the Q6 model will be served instead, and so on.

• 3b-homellm-v1: 3BV1
• 3b-homellm-v2: 3BV2
• 1b-homellm-v1: 1BV1