API Overview API Flow Errors Authentication Videos Collections Search Sound Align Sounds Speech Image Colors Faces Objects Text Motion Actions Scenes Concepts Related Concepts Text Sentiment
Dark mode Light mode
Dark mode Light mode

Welcome to the muse.ai APIs

Our APIs are fundamental to our video search service. By analyzing and splitting video into its constituent parts of motion, image, and sound, our Perceptual AIs can see, hear, and recognize a myriad of concepts. Because these capabilities can be employed in any service using digital media, they empower any basic to be part of the future of video.

Errors

Our APIs return a subset of basic HTTP response status codes. Additionally, the response body may include a JSON entry with a brief status string providing more details.

Status codes
200
OK

Everything went as expected.

400
Bad request

The request was not acceptable, often due to a missing parameter.

401
Unauthorized

No valid API key provided.

402
Payment required

Not enough credits.

404
Not found

The requested resource does not exist.

405
Method not allowed

The target resource doesn't support the method used.

429
Too many requests

Threshold limit has been reached.

500
Server error

Something went wrong on our end.

Example error
{
  "error": "file_not_provided"
}

Sound

Analyze the audio information of your files.

Image

The image APIs operate on individual images (or frames if you are analyzing a video). These tools rely on the color and spatial information.

Motion

Motion APIs analyze sequences of frames, and temporal as well as spatial patterns across them.

Concepts

Analyze the meaning of text.

API Flow

APIs that may require long-running jobs return a Job ID once a request is made. Using this Job ID you can retrieve the state of this job, and once it is complete (i.e. status: done), results can be fetched. The results are accessible for 1 hour after job completion.

1. Making an API Call

Request example
curl -X POST \
    -H "Key: YOUR_KEY" \
    -F "file=@input.wav" \
    https://api.muse.ai/speech
import requests

headers = {
    'Key': 'YOUR_KEY',
}

files = {
    'file': open('input.wav', 'rb'),
}

response = requests.post('https://api.muse.ai/speech', headers=headers, files=files)
const axios = require('axios');
const FormData = require('form-data');
const fs = require('fs');

const form = new FormData();
form.append('file', fs.readFileSync('input.wav'), 'input.wav');

const response = await axios.post(
    'https://api.muse.ai/speech',
    form,
    {
        headers: {
            ...form.getHeaders(),
            'Key': 'YOUR_KEY'
        }
    }
);
Response example
{
  "id": "72787541f3fc8170207ea446a22f60ce4130d950cbebcc058dd53bd321419beb"
}

2. Fetching results

Request example
curl -H "Key: YOUR_KEY" \
            https://api.muse.ai/speech/72787541f3fc8170207ea446a22f60ce4130d950cbebcc058dd53bd321419beb
import requests

headers = {
    'Key': 'YOUR_KEY',
}

response = requests.get('https://api.muse.ai/speech/72787541f3fc8170207ea446a22f60ce4130d950cbebcc058dd53bd321419beb', headers=headers)
const axios = require('axios');

const response = await axios.get('https://api.muse.ai/speech/72787541f3fc8170207ea446a22f60ce4130d950cbebcc058dd53bd321419beb', {
    headers: {
        'Key': 'YOUR_KEY'
    }
});
Response example (pending)
{
  "status": "pending"
}
Response example (done)
{
    "status": "done",
    "transcript": "we choose to go to the moon in this decade"
}

Authentication

In order to make requests to muse.ai APIs, you will need to use an authentication key. You can obtain a key in two ways: by using the Settings window in muse.ai application, or by making an authentication request using the API.

Using a key

To make requests to muse.ai APIs, pass the key using the Key header.

Creating a key

To create a key, make one of the following requests:

Using a password

Request example
curl -X POST \
     -H "Content-Type: application/json" \
     -d '{"email":"your-email@example.com","passwd":"your-password"}' \
     https://muse.ai/api/auth/login
import requests

headers = {
    # Already added when you pass json=
    # 'Content-Type': 'application/json',
}

json_data = {
    'email': 'your-email@example.com',
    'passwd': 'your-password',
}

response = requests.post('https://muse.ai/api/auth/login', headers=headers, json=json_data)
const axios = require('axios');

const response = await axios.post(
    'https://muse.ai/api/auth/login',
    {
        'email': 'your-email@example.com',
        'passwd': 'your-password'
    },
    {
        headers: {
            'Content-Type': 'application/json'
        }
    }
);
Show request parameter details Hide request parameter details
Request bodyapplication/json

email*

string

Email you used to sign up for muse.ai.

passwd*

string

Your muse.ai password.

* required

Response example
{
    "key": "2tHIVrn7GIghYCzVCvpgtf295d1e32d5"
}
Show response parameter details Hide response parameter details
Response body

keystring

Authentication key.

Using a key

Request example
curl -X POST \
     -H "Key: YOUR_KEY" \
     https://muse.ai/api/auth/keys
import requests

headers = {
    'Content-Type': 'application/json',
    'Key': 'YOUR_KEY',
}

response = requests.post('https://muse.ai/api/auth/keys', headers=headers)
const axios = require('axios');

const response = await axios.post(
    'https://muse.ai/api/auth/keys',
    '',
    {
        headers: {
            'Content-Type': 'application/json',
            'Key': 'YOUR_KEY'
        }
    }
);
Response example
{
    "key": "2tHIVrn7GIghYCzVCvpgtf295d1e32d5"
}
Show response parameter details Hide response parameter details
Response body

key

string

Authentication key.

Videos

Store and retrieve videos.

Upload

Request example
curl -X POST \
      -H "Key: YOUR_KEY" \
      -F "file=@The_Solar_System.mp4" \
      https://muse.ai/api/files/upload
import requests

headers = {
    'Key': 'YOUR_KEY',
}

files = {
    'file': open('The_Solar_System.mp4', 'rb'),
}

response = requests.post('https://muse.ai/api/files/upload', headers=headers, files=files)
const axios = require('axios');
const FormData = require('form-data');
const fs = require('fs');

const form = new FormData();
form.append('file', fs.readFileSync('hello_world.mp4'), 'hello_world.mp4');

const response = await axios.post(
    'https://muse.ai/api/files/upload',
    form,
    {
        headers: {
            ...form.getHeaders(),
            'Key': 'YOUR_KEY'
        }
    }
);
Show request parameter details Hide request parameter details
Request bodymultipart/form-data

file*

file

Video to store.

* required

Query parameters

collection

string

ID of a collection to add the video to after uploading.

visibility

string

Visibility of this video. One of "private", "hidden", "password", "unlisted", or "public" (default: private).

Info

Supported video formats

AVI, MOV, MP4, OGG, WMV, WEBM, MKV, 3GP, M4V, MPEG

Response example
{
    "fid": "b47a65b9351e38e6eb86780b09f3e9f9b8e9d795f1af5111f2f34a7f31f9e157",
    "svid": "8KsbyKv",
    "filename": "The_Solar_System.mp4",
    "title": "The Solar System",
    "description": "",
    "url": "https://cdn.muse.ai/w/b47a65b9351e38e6eb86780b09f3e9f9b8e9d795f1af5111f2f34a7f31f9e157/data",
    "duration": 426.84,
    "width": 1280,
    "height": 720,
    "size":251126000,
    "tcreated": 1559655942,
    "visibility": "public",
    "media": "video",
    "ingesting": false
}
Show response parameter details Hide response parameter details
Response body

fid

string

File ID.

svid

string

Video ID.

filename

string

Original filename.

title

string

Video title.

description

string

Video description.

url

string

Data URL.

duration

float

Video duration in seconds.

width

int

Width of the video in pixels.

height

int

Height of the video in pixels.

size

int

File size in bytes.

tcreated

int

Video uploaded timestamp (Unix time).

visibility

string

Visibility of this video. One of "private", "hidden", "password", "unlisted", or "public".

media

string

Media type. Either "video" or "audio".

ingesting

boolean

True if video is still being analyzed.

Cut a clip

Request example
curl -X POST \
      -H "Key: YOUR_KEY" \
      -F "svid=8KsbyKv" \
      -F "start=10" \
      -F "end=25" \
      https://muse.ai/api/files/cut
import requests

headers = {
    'Key': 'YOUR_KEY',
}

data = {
    'svid': '8KsbyKv',
    'start': 10,
    'end': 25,
}

response = requests.post('https://muse.ai/api/files/cut', headers=headers, data=data)
const axios = require('axios');
const FormData = require('form-data');

const form = new FormData();
form.append('svid', '8KsbyKv');
form.append('start', '10');
form.append('end', '25');

const response = await axios.post(
    'https://muse.ai/api/files/cut',
    form,
    {
        headers: {
            ...form.getHeaders(),
            'Key': 'YOUR_KEY'
        }
    }
);
Show request parameter details Hide request parameter details
Request body

svid*

string

Video ID of the source video to cut the clip from.

start*

float

Start time of the clip.

end*

float

End time of the clip.

titlestring

Video title (default: source video's title).

description

string

Video description (default: empty).

visibility

string

Visibility of the resulting video. One of "private", "hidden", "password", "unlisted", or "public" (default: source video's visibility).

* required

Response example
{
    "svid": "QB7jXla",
    "filename": "The_Solar_System.mp4",
    "title": "The Solar System",
    "description": "",
    "duration": 15.0,
    "width": 1280,
    "height": 720,
    "size": 538526,
    "tcreated": 1559656221,
    "visibility": "public",
    "ingesting": true,
    "cut_svid", "8KsbyKv",
    "cut_start", 10.0,
    "cut_end": 25.0
}
Show response parameter details Hide response parameter details
Response body

Response is the same as for the /upload endpoint. However, it won't include fid and url, because they're not available until ingestion is finished. Additionally includes, cut_svid, cut_start, and cut_end.

Update a video

Request example
curl -X POST \
    -H "Key: YOUR_KEY" \
    -H "Content-Type: application/json" \
    -d '{"title":"New Video Name","visibility":"public"}' \
    https://muse.ai/api/files/set/b47a65b9351e38e6eb86780b09f3e9f9b8e9d795f1af5111f2f34a7f31f9e157
import requests

      headers = {
          'Key': 'YOUR_KEY',
          # Already added when you pass json=
          # 'Content-Type': 'application/json',
      }

      json_data = {
          'title': 'New Video Name',
          'visibility': 'public',
      }

      response = requests.post('https://muse.ai/api/files/set/b47a65b9351e38e6eb86780b09f3e9f9b8e9d795f1af5111f2f34a7f31f9e157', headers=headers, json=json_data)
const axios = require('axios');

const response = await axios.post(
    'https://muse.ai/api/files/set/b47a65b9351e38e6eb86780b09f3e9f9b8e9d795f1af5111f2f34a7f31f9e157',
    {
        'title': 'New Video Name',
        'visibility': 'public'
    },
    {
        headers: {
            'Key': 'YOUR_KEY',
            'Content-Type': 'application/json'
        }
    }
);
Show request parameter details Hide request parameter details
Request bodyapplication/json

title

string

Video title.

description

string

Video description.

visibility

string

Video visibility. One of "private", "hidden", "password", "unlisted", or "public".

domains

list

A list of domains (strings) video embedding is limited to.

Delete a video

Request example
curl -X DELETE \
    -H "Key: YOUR_KEY" \
    https://muse.ai/api/files/delete/b47a65b9351e38e6eb86780b09f3e9f9b8e9d795f1af5111f2f34a7f31f9e157
import requests

headers = {
    'Key': 'YOUR_KEY',
}

response = requests.delete('https://muse.ai/api/files/delete/b47a65b9351e38e6eb86780b09f3e9f9b8e9d795f1af5111f2f34a7f31f9e157', headers=headers)
const axios = require('axios');

const response = await axios.delete('https://muse.ai/api/files/delete/b47a65b9351e38e6eb86780b09f3e9f9b8e9d795f1af5111f2f34a7f31f9e157', {
    headers: {
        'Key': 'YOUR_KEY'
    }
});

List videos

Request example
curl -H "Key: YOUR_KEY" \
          https://muse.ai/api/files/videos
import requests

headers = {
    'Key': 'YOUR_KEY',
}

response = requests.get('https://muse.ai/api/files/videos', headers=headers)
const axios = require('axios');

const response = await axios.get('https://muse.ai/api/files/videos', {
    headers: {
        'Key': 'YOUR_KEY'
    }
});
Response example
[
    {
        "fid": "b47a65b9351e38e6eb86780b09f3e9f9b8e9d795f1af5111f2f34a7f31f9e157",
        "svid": "8KsbyKv",
        "filename": "The_Solar_System.mp4",
        "title": "The Solar System",
        "description": "",
        "url": "https://cdn.muse.ai/w/b47a65b9351e38e6eb86780b09f3e9f9b8e9d795f1af5111f2f34a7f31f9e157/data",
        "duration": 426.84,
        "width": 1280,
        "height": 720,
        "size":251126000,
        "tcreated": 1559655942,
        "visibility": "public",
        "media": "video",
        "ingesting": false,
        "mature": false,
        "own": true,
        "views": 9716,
        "twatched": 3824799,
    }
]
Show response parameter details Hide response parameter details
Response body

fid

string

File ID.

svid

string

Video ID.

filename

string

Original filename.

title

string

Video title.

description

string

Video description.

url

string

Data URL.

duration

float

Video duration in seconds.

width

int

Width of the video in pixels.

height

int

Height of the video in pixels.

size

int

File size in bytes.

tcreated

int

Video uploaded timestamp (Unix time).

visibility

string

Video visibility. One of "private", "hidden", "password", "unlisted", or "public".

media

string

Media type. Either "video" or "audio".

ingesting

boolean

True if video is still being analyzed.

mature

boolean

True if video is flagged for mature content.

own

boolean

True if video is belongs to the owner of the key used.

views

int

View count.

twatched

int

Time watched in seconds.

Get a video

Request example
curl -H "Key: YOUR_KEY" \
      https://muse.ai/api/files/videos/8KsbyKv
import requests

headers = {
    'Key': 'YOUR_KEY',
}

response = requests.get('https://muse.ai/api/files/videos/8KsbyKv', headers=headers)
const axios = require('axios');

const response = await axios.get('https://muse.ai/api/files/videos/8KsbyKv', {
    headers: {
        'Key': 'YOUR_KEY'
    }
});
Response example
{
    "fid": "b47a65b9351e38e6eb86780b09f3e9f9b8e9d795f1af5111f2f34a7f31f9e157",
    "svid": "8KsbyKv",
    "filename": "The_Solar_System.mp4",
    "title": "The Solar System",
    "description": "",
    "url": "https://cdn.muse.ai/w/b47a65b9351e38e6eb86780b09f3e9f9b8e9d795f1af5111f2f34a7f31f9e157/data",
    "duration": 426.84,
    "width": 1280,
    "height": 720,
    "size":251126000,
    "tcreated": 1559655942,
    "visibility": "public",
    "media": "video",
    "ingesting": false,
    "mature": false,
    "own": true,
    "views": 9716,
    "twatched": 3824799,
}
Show response parameter details Hide response parameter details
Response body
See "List videos" for video data field documentation.

Update video cover / thumbnail

Using a timed frame from the video.
Request example
curl -X POST \
      -H "Key: YOUR_KEY" \
      https://muse.ai/api/files/set/b47a65b9351e38e6eb86780b09f3e9f9b8e9d795f1af5111f2f34a7f31f9e157/cover?t=time
import requests

headers = {
    'Key': 'YOUR_KEY',
}

params = {
    't': 'time',
}

response = requests.post('https://muse.ai/api/files/set/b47a65b9351e38e6eb86780b09f3e9f9b8e9d795f1af5111f2f34a7f31f9e157/cover', params=params, headers=headers)
const axios = require('axios');

const response = await axios.post(
    'https://muse.ai/api/files/set/b47a65b9351e38e6eb86780b09f3e9f9b8e9d795f1af5111f2f34a7f31f9e157/cover',
    '',
    {
        params: {
            't': 'time'
        },
        headers: {
            'Key': 'YOUR_KEY'
        }
    }
);
Show request parameter details Hide request parameter details
Query parameters

time*

float

Time of the video frame to set as thumbnail.

* required

Info

Supported formats

Int, Float

Response example
{
    "status": "success"
}
Uploading an image to set as the video thumbnail/cover.
Request example
curl -X POST \
      -H "Key: YOUR_KEY" \
      -F "file=@thumbnail.jpg" \
      https://muse.ai/api/files/set/b47a65b9351e38e6eb86780b09f3e9f9b8e9d795f1af5111f2f34a7f31f9e157/cover
import requests

headers = {
    'Key': 'YOUR_KEY',
}

files = {
    'file': open('thumbnail.jpg', 'rb'),
}

response = requests.post('https://muse.ai/api/files/set/b47a65b9351e38e6eb86780b09f3e9f9b8e9d795f1af5111f2f34a7f31f9e157/cover', headers=headers, files=files)
const axios = require('axios');
const FormData = require('form-data');
const fs = require('fs');

const form = new FormData();
form.append('file', fs.readFileSync('thumbnail.jpg'), 'thumbnail.jpg');

const response = await axios.post(
    'https://muse.ai/api/files/set/b47a65b9351e38e6eb86780b09f3e9f9b8e9d795f1af5111f2f34a7f31f9e157/cover',
    form,
    {
        headers: {
            ...form.getHeaders(),
            'Key': 'YOUR_KEY'
        }
    }
);
Show request parameter details Hide request parameter details
Request parameters

file*

file

Time of the video frame to set as thumbnail.

* required

Info

Supported formats

PNG, JPEG, JPG

Response example
{
    "status": "success"
}

Upload subtitles

Request example
curl -X POST \
    -H "Key: YOUR_KEY" \
    -F "subtitles=@Japanese.vtt" \
    https://muse.ai/api/files/subtitles/8KsbyKv
import requests

headers = {
    'Key': 'YOUR_KEY',
}

files = {
    'subtitles': open('Japanese.vtt', 'rb'),
}

response = requests.post('https://muse.ai/api/files/subtitles/8KsbyKv', headers=headers, files=files)
const axios = require('axios');
const FormData = require('form-data');
const fs = require('fs');

const form = new FormData();
form.append('subtitles', fs.readFileSync('Japanese.vtt'), 'Japanese.vtt');

const response = await axios.post(
    'https://muse.ai/api/files/subtitles/8KsbyKv',
    form,
    {
        headers: {
            ...form.getHeaders(),
            'Key': 'YOUR_KEY'
        }
    }
);
Show request parameter details Hide request parameter details
Request body

subtitles*

file

Subtitle file.

* required

Query parameters

name

string

Name of the subtitles.

Info

Supported subtitle formats

SRT, VTT

Response example
{
    "id": "N3wfN01lm",
    "name": "Japanese",
}

Update subtitles

Request example
curl -X POST \
-H "Key: YOUR_KEY" \
-F "name=Javanese" \
https://muse.ai/api/files/subtitles/8KsbyKv/N3wfN01lm/set
import requests

headers = {
    'Key': 'YOUR_KEY',
}

files = {
    'name': (None, 'Javanese'),
}

response = requests.post('https://muse.ai/api/files/subtitles/8KsbyKv/N3wfN01lm/set', headers=headers, files=files)
const axios = require('axios');
const FormData = require('form-data');

const form = new FormData();
form.append('name', 'Javanese');

const response = await axios.post(
    'https://muse.ai/api/files/subtitles/8KsbyKv/N3wfN01lm/set',
    form,
    {
        headers: {
            ...form.getHeaders(),
            'Key': 'YOUR_KEY'
        }
    }
);
Show request parameter details Hide request parameter details
Query parameters

name*

string

Name of the subtitles.

* required

Delete subtitles

Request example
curl -X POST \
      -H "Key: YOUR_KEY" \
      https://muse.ai/api/files/subtitles/8KsbyKv/N3wfN01lm/delete
import requests

headers = {
    'Key': 'YOUR_KEY',
}

response = requests.post('https://muse.ai/api/files/subtitles/8KsbyKv/N3wfN01lm/delete', headers=headers)
const axios = require('axios');

const response = await axios.post(
    'https://muse.ai/api/files/subtitles/8KsbyKv/N3wfN01lm/delete',
    '',
    {
        headers: {
            'Key': 'YOUR_KEY'
        }
    }
);

List subtitles

Request example
curl -H "Key: YOUR_KEY" \
        https://muse.ai/api/files/subtitles/8KsbyKv
import requests

headers = {
    'Key': 'YOUR_KEY',
}

response = requests.get('https://muse.ai/api/files/subtitles/8KsbyKv', headers=headers)
const axios = require('axios');

const response = await axios.get('https://muse.ai/api/files/subtitles/8KsbyKv', {
    headers: {
        'Key': 'YOUR_KEY'
    }
});
Response example
{
    "N3wfN01lm": "Javanese"
}

Download subtitles

Request example
curl -H "Key: YOUR_KEY" \
      https://muse.ai/api/files/subtitles/8KsbyKv/N3wfN01lm
import requests

headers = {
    'Key': 'YOUR_KEY',
}

response = requests.get('https://muse.ai/api/files/subtitles/8KsbyKv/N3wfN01lm', headers=headers)
const axios = require('axios');

const response = await axios.get('https://muse.ai/api/files/subtitles/8KsbyKv', {
    headers: {
        'Key': 'YOUR_KEY'
    }
});
const axios = require('axios');

const response = await axios.get('https://muse.ai/api/files/subtitles/8KsbyKv/N3wfN01lm', {
    headers: {
        'Key': 'YOUR_KEY'
    }
});

Collections

Store and retrieve collections.

List collections

Request example
curl -H "Key: YOUR_KEY" \
     https://muse.ai/api/files/collections
import requests

headers = {
    'Key': 'YOUR_KEY',
}

response = requests.get('https://muse.ai/api/files/collections', headers=headers)
fetch('https://muse.ai/api/files/collections', {
    headers: {
        'Key': 'YOUR_KEY'
    }
});
Response example
[
    {
        "name": "My Public Collection",
        "scid": "bdZofqF",
        "path": "",
        "tcreated": 1597247468,
        "visibility": "public",
        "videos": [
            {
                "svid": "8KsbyKv",
                "duration": 426.84,
                "fid": "b47a65b9351e38e6eb86780b09f3e9f9b8e9d795f1af5111f2f34a7f31f9e157",
                "url": "https://cdn.muse.ai/w/b47a65b9351e38e6eb86780b09f3e9f9b8e9d795f1af5111f2f34a7f31f9e157/data"
            }
        ]
    }
]
Show response parameter details Hide response parameter details
Response body

name

string

Collection name.

scid

string

Collection ID.

path

string

Parent collection scid (if "" then this is a root collection)

tcreated

int

Collection creation time (Unix time).

visibility

string

Collection visibility.

videos

list

A list of minimal descriptions of videos contained in this collection.

Get a collection

Request example
curl -H "Key: YOUR_KEY" \
     https://muse.ai/api/files/collections/bdZofqF
import requests

headers = {
    'Key': 'YOUR_KEY',
}

response = requests.get('https://muse.ai/api/files/collections/bdZofqF', headers=headers)
const axios = require('axios');

const response = await axios.get('https://muse.ai/api/files/collections/bdZofqF', {
    headers: {
        'Key': 'YOUR_KEY'
    }
});
Response example
{
    "name": "My Public Collection",
    "own": true,
    "path": "",
    "visibility": "public",
    "videos": [
        {
            "svid": "8KsbyKv",
            "description": "",
            "duration": 426.84,
            "fid": "b47a65b9351e38e6eb86780b09f3e9f9b8e9d795f1af5111f2f34a7f31f9e157",
            "filename": "The_Solar_System.mp4",
            "ingesting": false,
            "size":251126000,
            "tadded": 1597247477,
            "tcreated": 1559655942,
            "title": "The Solar System",
            "url": "https://cdn.mus.ai/w/b47a65b9351e38e6eb86780b09f3e9f9b8e9d795f1af5111f2f34a7f31f9e157/data",
            "views": 0,
            "visibility": "public",
            "media": "video",
            "width": 1920,
            "height": 1080
        }
    ]
}
Show response parameter details Hide response parameter details
Response body

name

string

Collection name.

own

boolean

Indicates whether the collection is owned by you.

path

string

Parent collection scid (if "" then this is a root collection)

visibility

string

Collection visibility.

videos

list

A list of videos contained in this collection (see "List videos").

Create a collection

Request example
curl -X POST \
     -H "Key: YOUR_KEY" \
     -H "Content-Type: application/json" \
     -d '{"name":"My Public Collection","visibility":"public"}' \
     https://muse.ai/api/files/collections
import requests

headers = {
    'Key': 'YOUR_KEY',
}

json_data = {
    'name': 'My Public Collection',
    'visibility': 'public',
}

response = requests.post('https://muse.ai/api/files/collections', headers=headers, json=json_data)
const axios = require('axios');

const response = await axios.post(
    'https://muse.ai/api/files/collections',
    {
        'name': 'My Public Collection',
        'visibility': 'public'
    },
    {
        headers: {
            'Key': 'YOUR_KEY',
            'Content-Type': 'application/json'
        }
    }
);
Show request parameter details Hide request parameter details
Request bodyapplication/json

name*

string

Collection name.

visibility*

string

Collection visibility. One of "private", "hidden", "password", "unlisted", or "public".

parent

string

Parent collection ID.

* required

Response example
{
    "scid": "8KsbyKv"
}

Delete a collection

Request example
curl -X DELETE \
     -H "Key: YOUR_KEY" \
     https://muse.ai/api/files/collections/8KsbyKv
import requests

headers = {
    'Key': 'YOUR_KEY',
}

response = requests.delete('https://muse.ai/api/files/collections/8KsbyKv', headers=headers)
const axios = require('axios');

const response = await axios.delete('https://muse.ai/api/files/collections/8KsbyKv', {
    headers: {
        'Key': 'YOUR_KEY'
    }
});

Move a collection

Request example
curl -X POST \
     -H "Key: YOUR_KEY" \
     -H "Content-Type: application/json" \
     -d '{"scid":"8KsbyKv"}' \
     https://muse.ai/api/files/collections/bdZofqF/move
import requests

headers = {
    'Key': 'YOUR_KEY',
}

json_data = {
    'scid': '8KsbyKv',
}

response = requests.post('https://muse.ai/api/files/collections/bdZofqF/move', headers=headers, json=json_data)
const axios = require('axios');

const response = await axios.post(
    'https://muse.ai/api/files/collections/bdZofqF/move',
    {
        'svid': '8KsbyKv',
    },
    {
        headers: {
            'Key': 'YOUR_KEY',
            'Content-Type': 'application/json'
        }
    }
);
Show request parameter details Hide request parameter details
Request bodyapplication/json

scid*

string

Parent collection ID.

* required

Response example
{
    "path": "8KsbyKv"
}

Add a video to a collection

Request example
curl -X POST \
     -H "Key: YOUR_KEY" \
     -H "Content-Type: application/json" \
     -d '{"svid":"8KsbyKv"}' \
     https://muse.ai/api/files/collections/bdZofqF/add
import requests

headers = {
    'Key': 'YOUR_KEY',
}

json_data = {
    'svid': '8KsbyKv',
}

response = requests.post('https://muse.ai/api/files/collections/bdZofqF/add', headers=headers, json=json_data)
const axios = require('axios');

const response = await axios.post(
    'https://muse.ai/api/files/collections/bdZofqF/add',
    {
        'svid': '8KsbyKv',
    },
    {
        headers: {
            'Key': 'YOUR_KEY',
            'Content-Type': 'application/json'
        }
    }
);
Show request parameter details Hide request parameter details
Request bodyapplication/json

svid*

string

Video ID.

* required

Remove a video from a collection

Request example
curl -X DELETE \
     -H "Key: YOUR_KEY" \
     https://muse.ai/api/files/collections/bdZofqF/8KsbyKv
import requests

headers = {
    'Key': 'YOUR_KEY',
}

response = requests.delete('https://muse.ai/api/files/collections/bdZofqF/8KsbyKv', headers=headers)
const axios = require('axios');

const response = await axios.delete(
    'https://muse.ai/api/files/collections/bdZofqF/8KsbyKv',
    {
        headers: {
            'Key': 'YOUR_KEY',
        }
    }
);

Speech

Open demo

Perform speech recognition on video or audio files to generate transcripts of speech.

Transcribe audio

Request example
curl -X POST \
        -H "Key: YOUR_KEY" \
        -F "file=@input.wav" \
        https://api.muse.ai/speech
import requests

headers = {
    'Key': 'YOUR_KEY',
}

files = {
    'file': open('input.wav', 'rb'),
}

response = requests.post('https://api.muse.ai/speech', headers=headers, files=files)
const axios = require('axios');
const FormData = require('form-data');
const fs = require('fs');

const form = new FormData();
form.append('file', fs.readFileSync('input.wav'), 'input.wav');

const response = await axios.post(
    'https://api.muse.ai/speech',
    form,
    {
        headers: {
            ...form.getHeaders(),
            'Key': 'YOUR_KEY'
        }
    }
);

* Follows the API Flow steps

Show request parameter details Hide request parameter details
Request body

file*

file

File to process

* required

Info

Supported video formats

MP4, MP3, WAV, FLAC

Max file size

5GB

Max duration

5 hours

Response example
{
  "status": "done",
  "transcript": "we choose to go to the moon in this decade"
}
Show response parameter details Hide response parameter details
Response body

status

string

Indicates the status of the job ("done", "pending", or "error")

transcript

string

Transcript of the provided audio

Align

Open demo

Find a position of each word in a speech segment by aligning transcript with audio.

Align audio with transcript

Request example
curl -X POST \
        -H "Key: YOUR_KEY" \
        -F "file=@input.wav" \
        -F "phrase=All men are created equal. I have a dream." \
        https://api.muse.ai/align
import requests

headers = {
    'Key': 'YOUR_KEY',
}

files = {
    'file': open('input.wav', 'rb'),
    'phrase': (None, 'All men are created equal. I have a dream.'),
}

response = requests.post('https://api.muse.ai/align', headers=headers, files=files)
const axios = require('axios');
const FormData = require('form-data');
const fs = require('fs');

const form = new FormData();
form.append('file', fs.readFileSync('input.wav'), 'input.wav');
form.append('phrase', 'All men are created equal. I have a dream.');

const response = await axios.post(
    'https://api.muse.ai/align',
    form,
    {
        headers: {
            ...form.getHeaders(),
            'Key': 'YOUR_KEY'
        }
    }
);

* Follows the API Flow steps

Show request parameter details Hide request parameter details
Request bodymultipart/form-data

file*

file

File to process

phrase*

string

Phrase to align

file*

file

Archive containing the audio file and .txt with the phrase to align

* required

Info

Supported video formats

MP4, MP3, WAV, FLAC

Max file size

50MB

Max duration

1 minute

Response example
{
  "status": "done",
  "words": [[0.44, 0.71, "All"], [0.71, 0.97, "men"], [0.97, 1.2, "are"], [1.2, 1.59, "created"], [1.59, 2.13, "equal"], [2.13, 2.38, "I"], [2.38, 2.6, "have"], [2.6, 2.69, "a"], [2.69, 3.26, "dream"]]
}
Show response parameter details Hide response parameter details
Response body

status

string

Indicates the status of the job ("done", "pending", or "error")

words

array

A list of [start, end, word] triples

Sounds

Open demo

Identify sound types in an audio file. Currently supported types include: speech, music, applause, laughter, and silence.

Identify audio

Request example
curl -X POST \
            -H "Key: YOUR_KEY" \
            -F "file=@input.wav" \
            https://api.muse.ai/sounds
import requests

headers = {
    'Key': 'YOUR_KEY',
}

files = {
    'file': open('input.wav', 'rb'),
}

response = requests.post('https://api.muse.ai/sounds', headers=headers, files=files)
const axios = require('axios');
const FormData = require('form-data');
const fs = require('fs');

const form = new FormData();
form.append('file', fs.readFileSync('input.wav'), 'input.wav');

const response = await axios.post(
    'https://api.muse.ai/sounds',
    form,
    {
        headers: {
            ...form.getHeaders(),
            'Key': 'YOUR_KEY'
        }
    }
);

* Follows the API Flow steps

Show request parameter details Hide request parameter details
Request body

file*

file

File to process

* required

Info

Supported video formats

MP4, MP3, WAV, FLAC

Max file size

5GB

Max duration

5 hours

Response example
{
  "status": "done",
  "sounds": {
    "speech": [[0.53, 0.87], [2.02, 3.09]],
    "music": [[0.87, 1.3]],
    "silence": [[1.3, 2.31]],
    "laughter": [[2.65, 2.92]],
    "applause": [[2.31, 2.65]]
  }
}
Show response parameter details Hide response parameter details
Response body

status

string

Indicates the status of the job ("done", "pending", or "error")

soundsmap

Key is a string with the sound type and the value is a list of pairs of start and end times

Faces

Open demo

Detect when and where faces appear in a video. Similar faces are grouped together, and known faces are labeled.

Detect faces

Request example
curl -X POST \
            -H "Key: YOUR_KEY" \
            -F "file=@input.mp4" \
            https://api.muse.ai/faces
import requests

headers = {
    'Key': 'YOUR_KEY',
}

files = {
    'file': open('input.mp4', 'rb'),
}

response = requests.post('https://api.muse.ai/faces', headers=headers, files=files)
const axios = require('axios');
const FormData = require('form-data');
const fs = require('fs');

const form = new FormData();
form.append('file', fs.readFileSync('input.wav'), 'input.wav');

const response = await axios.post(
    'https://api.muse.ai/sounds',
    form,
    {
        headers: {
            ...form.getHeaders(),
            'Key': 'YOUR_KEY'
        }
    }
);

* Follows the API Flow steps

Show request parameter details Hide request parameter details
Request body

file*

file

File to process

* required

Info

Supported video formats

AVI, MP4, MKV, MOV, WEBM

Max file size

5GB

Response example
{
    "status": "done",
    "faces": {
      "e5d07990e0ad443d90a8ca4ef11cf3ba": {
        "label": null,
        "faces": {"95dbaa6c00b09be87fb4b2a038cc509036cf59fdc93cd2b8445f755203a990a3": {
          "s": 2.12,
          "e": 5.32,	  
          "x": 0.48,
          "y": 0.36,
          "w": 0.15,
          "h": 0.17,
          "q": 0.5 
        }
      }
    }
  }
}
Show response parameter details Hide response parameter details
Response body

status

string

Indicates the status of the job ("done", "pending", or "error")

faces

map

Key is a cluster id and value is a map representing the cluster

Show item members

List faces

Request example
curl -H "Key: YOUR_KEY" \
          https://api.muse.ai/faces/db
import requests

headers = {
    'Key': 'YOUR_KEY',
}

response = requests.get('https://api.muse.ai/faces/db', headers=headers)
const axios = require('axios');

const response = await axios.get('https://api.muse.ai/faces/db', {
    headers: {
        'Key': 'YOUR_KEY'
    }
});
Response example
{
  "detected":
    {"815b4f79c996408abac72ca6866da1d7":
      {"label": null,
       "faces":
        {"fdeed8e00e5d651f737e90f343e175080244c040e0f1518b30654baebf3bef3a":
          {"moment": {
            "f": "4c2ec982c22bbb23fdc734762d7310230e09bec386fbac7946a5a6453eb16e0b",
            "s": 174.0,
            "e": 180.92,
            "x": 0.28,
            "y": 0.14,
            "w": 0.21,
            "h": 0.49,
            "q": 0.5}
          }
        }
      }
    },
  "deleted":
    {"142abde00e5d651f737e90f343e175080244c040e0f1518b30654baebf3bef3a":
      {"moment": {
        "f": "4c2ec982c22bbb23fdc734762d7310230e09bec386fbac7946a5a6453eb16e0b",
        "s": 108.0,
        "e": 108.75,
        "x": 0.74,
        "y": 0.23,
        "w": 0.13,
        "h": 0.26,
        "q": 1}
      }
    },
  "ignored":
    {"504afa20acda45ec4d690dc7aa60f612e90b0954b82a3eac369964a31d62ed4d":
      {"moment": {
        "f": "4c2ec982c22bbb23fdc734762d7310230e09bec386fbac7946a5a6453eb16e0b",
        "s": 76.0,
        "e": 77.58,
        "x": 0.19,
        "y": 0.4,
        "w": 0.08,
        "h": 0.17,
        "q": 1}
      }
    }
}
Show response parameter details Hide response parameter details
Response body

detectedmap

List of detected faces, organized into clusters, where the key is a cluster id and value is a map representing the cluster

Show item members

ignored

map

List of faces which will not be detected in future videos, unlabeled and not organized into clusters

deleted

map

List of faces which have been deleted from the list of detected faces, unlabeled and not organized into clusters

Modify faces

Request example
curl -X POST \
        -H "Key: YOUR_KEY" \
        -H "Content-Type: application/json" \
        -d '{"0c3e64b7c7fa9753b589d90a1e0c821fa76c949256614d31bb0cd3ce6346d1c1":
            {"label": "Jenny",
            "delete": False,
            "ignore": False,
            "purge": False}}' \
        https://api.muse.ai/faces/modify
import requests

headers = {
    'Key': 'YOUR_KEY',
    'Content-Type': 'application/json',
}

data = '{"0c3e64b7c7fa9753b589d90a1e0c821fa76c949256614d31bb0cd3ce6346d1c1":\n            {"label": "Jenny",\n            "delete": False,\n            "ignore": False,\n            "purge": False}}'

response = requests.post('https://api.muse.ai/faces/modify', headers=headers, data=data)
const axios = require('axios');

const response = await axios.post(
    'https://api.muse.ai/faces/modify',
    '{"0c3e64b7c7fa9753b589d90a1e0c821fa76c949256614d31bb0cd3ce6346d1c1":\n            {"label": "Jenny",\n            "delete": False,\n            "ignore": False,\n            "purge": False}}',
    {
        headers: {
            'Key': 'YOUR_KEY',
            'Content-Type': 'application/json'
        }
    }
);
Show request parameter details Hide request parameter details
Request body

json*

data

an object containing face IDs and corresponding changes

Show item members

* required

Response example
{
  "status": "ok"
}
Show response parameter details Hide response parameter details
Response body

status

string

Indicates the success of the job (expected result: "ok")

Objects

Open demo

Recognize common objects in images.

Identify objects in an image

Request example
curl -X POST \
        -H "Key: YOUR_KEY" \
        -F "file=@input.png" \
        https://api.muse.ai/objects
import requests

headers = {
    'Key': 'YOUR_KEY',
}

files = {
    'file': open('input.png', 'rb'),
}

response = requests.post('https://api.muse.ai/objects', headers=headers, files=files)
const axios = require('axios');
const FormData = require('form-data');
const fs = require('fs');

const form = new FormData();
form.append('file', fs.readFileSync('input.png'), 'input.png');

const response = await axios.post(
    'https://api.muse.ai/objects',
    form,
    {
        headers: {
            ...form.getHeaders(),
            'Key': 'YOUR_KEY'
        }
    }
);

* Follows the API Flow steps

Show request parameter details Hide request parameter details
Request body

file*

file

File to process

* required

Info

Supported video formats

JPG, JPEG, PNG, GIF, BMP, WEBP

Max file size

5GB

Response example
{
  "status": "done",
  "objects": {
    "person": [{"x": 0.508, "y": 0.281, "w": 0.514, "h": 0.233, "q": 0.5},
               {"x": 0.425, "y": 0.102, "w": 0.310, "h": 0.410, "q": 0.5}],
    "bicycle": [{"x": 0.488, "y": 0.402, "w": 0.491, "h": 0.221, "q": 0.5}]
  }
}
Show response parameter details Hide response parameter details
Response body

status

string

Indicates the status of the job ("done", "pending", or "error")

objects

map

Key is detected objects and value is their coordinates

Show item members

Coordinate representation

Actions

Open demo

Recognize actions such as dancing, flying, driving etc. in a video.

Recognize actions

Request example
curl -X POST \
        -H "Key: YOUR_KEY" \
        -F "file=@input.mp4" \
        https://api.muse.ai/actions
import requests

headers = {
    'Key': 'YOUR_KEY',
}

files = {
    'file': open('input.mp4', 'rb'),
}

response = requests.post('https://api.muse.ai/actions', headers=headers, files=files)
const axios = require('axios');
const FormData = require('form-data');
const fs = require('fs');

const form = new FormData();
form.append('file', fs.readFileSync('input.mp4'), 'input.mp4');

const response = await axios.post(
    'https://api.muse.ai/actions',
    form,
    {
        headers: {
            ...form.getHeaders(),
            'Key': 'YOUR_KEY'
        }
    }
);

* Follows the API Flow steps

Show request parameter details Hide request parameter details
Request body

file*file

File to process

* required

Info

Supported video formats

AVI, MP4, MKV, MOV, WEBM

Max file size

5GB

Max duration

1 hour

Response example
{
  "status": done,
  "actions": {
    "driving": [{"s": 5.62, "e": 6.28, "q": 0.5}, {"s": 14.83, "e": 15.46, "q": 0.5}],
    "running": [{"s": 82.29, "e": 82.93, "q": 0.5}],
    "riding": [{"s": 107.8, "e": 108.44, "q": 0.5}]
  }
}
Show response parameter details Hide response parameter details
Response body

status

string

Indicates the status of the job ("done", "pending", or "error")

actions

map

Key is detected action and value is a list with its occurrences

Show item members

Text

Open demo

Find and recognize text in an image.

Identify text in an image

Request example
curl -X POST \
        -H "Key: YOUR_KEY" \
        -F "file=@input.png" \
        https://api.muse.ai/text
import requests

headers = {
    'Key': 'YOUR_KEY',
}

files = {
    'file': open('input.png', 'rb'),
}

response = requests.post('https://api.muse.ai/text', headers=headers, files=files)
const axios = require('axios');
const FormData = require('form-data');
const fs = require('fs');

const form = new FormData();
form.append('file', fs.readFileSync('input.png'), 'input.png');

const response = await axios.post(
    'https://api.muse.ai/text',
    form,
    {
        headers: {
            ...form.getHeaders(),
            'Key': 'YOUR_KEY'
        }
    }
);

* Follows the API Flow steps

Show request parameter details Hide request parameter details
Request body

file*

file

File to process

* required

Info

Supported formats

JPEG, PNG, BMP, WEBP

Max file size

50MB

Response example
{
  "status": "done",
  "text": {
    "hello": [{"x": 0.108, "y": 0.281, "w": 0.314, "h": 0.233, "q": 0.5}],
    "world": [{"x": 0.452, "y": 0.280, "w": 0.295, "h": 0.221, "q": 0.5}]
  }
}
Show response parameter details Hide response parameter details
Response body

status

string

Indicates the status of the job ("done", "pending", or "error")

text

map

Key is a string with the text and the value is the bounding box

Show item members

Coordinate representation

Scenes

Open demo

Perform splitting of videos based on shot boundaries.

Detect and cluster scenes

Request example
curl -X POST \
        -H "Key: YOUR_KEY" \
        -F "file=@input.mp4" \
        https://api.muse.ai/scenes
import requests

headers = {
    'Key': 'YOUR_KEY',
}

files = {
    'file': open('input.mp4', 'rb'),
}

response = requests.post('https://api.muse.ai/scenes', headers=headers, files=files)
const axios = require('axios');
const FormData = require('form-data');
const fs = require('fs');

const form = new FormData();
form.append('file', fs.readFileSync('input.mp4'), 'input.mp4');

const response = await axios.post(
    'https://api.muse.ai/scenes',
    form,
    {
        headers: {
            ...form.getHeaders(),
            'Key': 'YOUR_KEY'
        }
    }
);

* Follows the API Flow steps

Show request parameter details Hide request parameter details
Request body

file*

file

File to process

* required

Info

Supported video formats

AVI, MP4, MKV, MOV, WEBM

Max file size

5GB

Max duration

5 hours

Response example
{
  "status": "done",
  "scenes": [[[0.0, 8.25]], [[8.28, 16.41], [16.45, 18.92]],
             [[18.77, 25.22]], [[25.26, 45.30]]]
}
Show response parameter details Hide response parameter details
Response body

status

string

Indicates the status of the job ("done", "pending", or "error")

scenes

array

List of scene clusters containing one or more scenes, represented by [start, end] times in seconds

Colors

Open demo

Identify colors in images.

Identify colors

Request example
curl -X POST \
     -H "Key: YOUR_KEY" \
     -F "file=@input.png" \
     https://api.muse.ai/colors
import requests

headers = {
    'Key': 'YOUR_KEY',
}

files = {
    'file': open('input.png', 'rb'),
}

response = requests.post('https://api.muse.ai/colors', headers=headers, files=files)
const axios = require('axios');
const FormData = require('form-data');
const fs = require('fs');

const form = new FormData();
form.append('file', fs.readFileSync('input.png'), 'input.png');

const response = await axios.post(
    'https://api.muse.ai/colors',
    form,
    {
        headers: {
            ...form.getHeaders(),
            'Key': 'YOUR_KEY'
        }
    }
);

* Follows the API Flow steps

Show request parameter details Hide request parameter details
Request body

file*

file

File to process

* required

Info

Supported video formats

JPEG, PNG, BMP, WEBP

Max file size

50MB

Response example
{
  "status": "done",
  "colors": {"black": 0.252, "blue": 0.035, "brown": 0.08, "green": 0.01, "grey": 0.591, "orange": 0.01, "pink": 0.01, "purple": 0.0, "red": 0.0, "white": 0.01, "yellow": 0.0}
}
Show response parameter details Hide response parameter details
Response body

status

string

Indicates the status of the job ("done", "pending", or "error")

colorsmap

Key is a string with the color and the value is the proportion of that color in the image

Sense

Open demo

Detect how positive/negative the text is.

Sentiment analysis

Request example
curl -X POST \
        -H "Key: YOUR_KEY" \
        -d "this is cool and awesome and not bad" \
        https://api.muse.ai/sense
import requests

headers = {
    'Key': 'YOUR_KEY',
    'Content-Type': 'application/x-www-form-urlencoded',
}

data = 'this is cool and awesome and not bad'

response = requests.post('https://api.muse.ai/sense', headers=headers, data=data)
const axios = require('axios');

const response = await axios.post(
    'https://api.muse.ai/sense',
    'this is cool and awesome and not bad',
    {
        headers: {
            'Key': 'YOUR_KEY',
            'Content-Type': 'application/x-www-form-urlencoded'
        }
    }
);
Show request parameter details Hide request parameter details
Request bodytext/plain

text*

string

Text to analyze

* required

Info

Max file size

5MB

Response example
{
  "positive": 8,
  "negative": 0,
  "total": 8
}
Show response parameter details Hide response parameter details
Response body

positive

int

Positivity estimate for the text.

negative

int

Negativity estimate for the text.

total

int

Word count in the text.