Dark mode Light mode

Welcome to the muse.ai APIs

Our APIs are fundamental to our video search service. By analyzing and splitting video into its constituent parts of motion, image, and sound, our Perceptual AIs can see, hear, and recognize a myriad of concepts. Because these capabilities can be employed in any service using digital media, they empower any visionary to be part of the future of video.

Errors

Our APIs return a subset of basic HTTP response status codes. Additionally, the response body may include a JSON entry with a brief status string providing more details.

Status codes
200
OK

Everything went as expected.

400
Bad request

The request was not acceptable, often due to a missing parameter.

401
Unauthorized

No valid API key provided.

402
Payment required

Not enough credits.

404
Not found

The requested resource does not exist.

429
Too many requests

Threshold limit has been reached.

500
Server error

Something went wrong on our end.

Example error
{
  "error": "file_not_provided"
}

Sound

Analyze the audio information of your files.

Image

The image APIs operate on individual images (or frames if you are analyzing a video). These tools rely on the color and spatial information.

Motion

Motion APIs analyze sequences of frames, and temporal as well as spatial patterns across them.

Concepts

Analyze the meaning of text.

API Flow

APIs that may require long-running jobs return a Job ID once a request is made. Using this Job ID you can retrieve the state of this job, and once it is complete (i.e. status: done), results can be fetched. The results are accessible for 1 hour after job completion.

1. Making an API Call

Request example
curl -X POST \
     -H "Key: YOUR_KEY" \
     -F "file=@input.png" \
     https://api.muse.ai/objects
Response example
{
  "id": "72787541f3fc8170207ea446a22f60ce4130d950cbebcc058dd53bd321419beb"
}

2. Fetching results

Request example
CURL -H "Key: YOUR_KEY" \
     https://api.muse.ai/objects/72787541f3fc8170207ea446a22f60ce4130d950cbebcc058dd53bd321419beb         
Response example (pending)
{
  "status": "pending"
}
Response example (done)
{
  "status": "done",
  "objects": {
      "bicycle": [{"x" : 0.0067, "y" : 0.3212, "w" : 0.2299, "h" : 0.1837, "q": 0.5},
                  {"x" : 0.1715, "y" : 0.3966, "w" : 0.6977, "h" : 0.5276, "q': 0.5}],
      "dog": [{"x" : 0.3205, "y" : 0.096, "w" : 0.4474, "h" : 0.4667, "q': 0.5}]
  }
}

Authentication

In order to make requests to muse.ai APIs, you will need to use an authentication key. You can obtain a key in two ways: by using the Settings window in muse.ai application, or by making an authentication request using the API.

Using a key

To make requests to muse.ai APIs, pass the key using the Key header.

Creating a key

To create a key, make one of the following requests:

Request example
curl -X POST \
     -H "Content-Type: application/json" \
     -d '{"email":"your-email@example.com","passwd":"your-password"}' \
     https://muse.ai/api/auth/login
Show request parameter details Hide request parameter details
Request bodyapplication/json

email*string

Email you used to sign up for muse.ai

passwd*string

Your muse.ai password

* required

Response example
{
    "key": "2tHIVrn7GIghYCzVCvpgtf295d1e32d5"
}
Request example
curl -X POST \
     -H "Content-Type: application/json" \
     -H "Key: YOUR_KEY" \
     https://muse.ai/api/auth/keys
Response example
{
    "key": "2tHIVrn7GIghYCzVCvpgtf295d1e32d5"
}

Storage

Store and retrieve videos.

Upload

Request example
curl -X POST \
     -H "Key: YOUR_KEY" \
     -F "file=@hello_world.mp4" \
     https://muse.ai/api/files/upload
Show request parameter details Hide request parameter details
Request body

file*file

Video to store

* required

Info

Supported video formats

AVI, MOV, MP4, OGG, WMV, WEBM, MKV, 3GP, M4V, MPEG

Response example
{
    "fid": "66631dd21830e71422ac52389a21a61c9659a6aaf97018ebd3aee4a8602002e6",
    "svid": "aPEHdtr",
    "filename": "hello_world.mp4",
    "title": "hello world",
    "description": "",
    "url": "https://vv2-0-0.muse.ai/v/66631dd21830e71422ac52389a21a61c9659a6aaf97018ebd3aee4a8602002e6/data",
    "duration": 167.741,
    "width": 1280,
    "height": 720,
    "size": 30228172,
    "tcreated": 1559655942,
    "visibility": "private",
    "ingesting": false
}
Show response parameter details Hide response parameter details
Response body

fidstring

File id

svidstring

Human readable file id

filenamestring

Original filename

titlestring

User-editable title

descriptionstring

User-editable description

urlstring

Direct URL for video streaming

durationfloat

Video duration in seconds

widthint

Width of the video in pixels

heightint

Height of the video in pixels

sizeint

File size in bytes

tcreatedint

Video uploaded timestamp (Unix time)

visibilitystring

"private" if private and "public" if public

ingestingboolean

True if video is still being analyzed

Delete video

Request example
curl -X DELETE \
     -H "Key: YOUR_KEY" \
     https://muse.ai/api/files/delete/66631dd21830e71422ac52389a21a61c9659a6aaf97018ebd3aee4a8602002e6

List videos

Request example
curl -H "Key: YOUR_KEY" \
     https://muse.ai/api/files/list
Response example
[
    {
        "fid": "66631dd21830e71422ac52389a21a61c9659a6aaf97018ebd3aee4a8602002e6",
        "svid": "aPEHdtr",
        "filename": "hello_world.mp4",
        "title": "hello world",
        "description": "",
        "url": "https://vv2-0-0.muse.ai/v/66631dd21830e71422ac52389a21a61c9659a6aaf97018ebd3aee4a8602002e6/data",
        "duration": 167.741,
        "width": 1280,
        "height": 720,
        "size": 30228172,
        "tcreated": 1559655942,
        "visibility": "private",
        "ingesting": false
    }
]
Show response parameter details Hide response parameter details
Response body

fidstring

File id

svidstring

Human readable file id

filenamestring

Original filename

titlestring

User-editable title

descriptionstring

User-editable description

urlstring

Direct URL for video streaming

durationfloat

Video duration in seconds

widthint

Width of the video in pixels

heightint

Height of the video in pixels

sizeint

File size in bytes

tcreatedint

Video uploaded timestamp (Unix time)

visibilitystring

"private" if private and "public" if public

ingestingboolean

True if video is still being analyzed

Speech

Perform speech recognition on video or audio files to generate transcripts of speech.

Transcribe audio

Request example
curl -X POST \
     -H "Key: YOUR_KEY" \
     -F "file=@input.wav" \
     https://api.muse.ai/speech

* Follows the API Flow steps

Show request parameter details Hide request parameter details
Request body

file*file

File to process

* required

Info

Supported video formats

MP4, MP3, WAV, FLAC

Max file size

5GB

Max duration

5 hours

Response example
{
  "status": "done",
  "transcript": "we choose to go to the moon in this decade"
}
Show response parameter details Hide response parameter details
Response body

statusstring

Indicates the status of the job ("done", "pending", or "error")

transcriptstring

Transcript of the provided audio

Align

Find a position of each word in a speech segment by aligning transcript with audio.

Align audio with transcript

Request example
curl -X POST \
     -H "Key: YOUR_KEY" \
     -F "file=@input.wav" \
     -F "phrase=All men are created equal. I have a dream." \
     https://api.muse.ai/align

* Follows the API Flow steps

Show request parameter details Hide request parameter details
Request bodymultipart/form-data

file*file

File to process

phrase*string

Phrase to align

file*file

Archive containing the audio file and .txt with the phrase to align

* required

Info

Supported video formats

MP4, MP3, WAV, FLAC

Max file size

50MB

Max duration

1 minute

Response example
{
  "status": "done",
  "words": [[0.44, 0.71, "All"], [0.71, 0.97, "men"], [0.97, 1.2, "are"], [1.2, 1.59, "created"], [1.59, 2.13, "equal"], [2.13, 2.38, "I"], [2.38, 2.6, "have"], [2.6, 2.69, "a"], [2.69, 3.26, "dream"]]
}
Show response parameter details Hide response parameter details
Response body

statusstring

Indicates the status of the job ("done", "pending", or "error")

wordsarray

A list of [start, end, word] triples

Sounds

Identify sound types in an audio file. Currently supported types include: speech, music, applause, laughter, and silence.

Identify audio

Request example
curl -X POST \
     -H "Key: YOUR_KEY" \
     -F "file=@input.wav" \
     https://api.muse.ai/sounds

* Follows the API Flow steps

Show request parameter details Hide request parameter details
Request body

file*file

File to process

* required

Info

Supported video formats

MP4, MP3, WAV, FLAC

Max file size

5GB

Max duration

5 hours

Response example
{
  "status": "done",
  "sounds": {
    "speech": [[0.53, 0.87], [2.02, 3.09]],
    "music": [[0.87, 1.3]],
    "silence": [[1.3, 2.31]],
    "laughter": [[2.65, 2.92]],
    "applause": [[2.31, 2.65]]
  }
}
Show response parameter details Hide response parameter details
Response body

statusstring

Indicates the status of the job ("done", "pending", or "error")

soundsmap

Key is a string with the sound type and the value is a list of pairs of start and end times

Faces

Detect when and where faces appear in a video. Similar faces are grouped together, and known faces are labeled.

Detect faces

Request example
curl -X POST \
     -H "Key: YOUR_KEY" \
     -F "file=@input.mp4" \
     https://api.muse.ai/faces

* Follows the API Flow steps

Show request parameter details Hide request parameter details
Request body

file*file

File to process

* required

Info

Supported video formats

AVI, MP4, MKV, MOV, WEBM

Max file size

5GB

Response example
{
    "status": "done",
    "faces": {
      "e5d07990e0ad443d90a8ca4ef11cf3ba": {
        "label": null,
        "faces": {"95dbaa6c00b09be87fb4b2a038cc509036cf59fdc93cd2b8445f755203a990a3": {
          "s": 2.12,
          "e": 5.32,	  
          "x": 0.48,
          "y": 0.36,
          "w": 0.15,
          "h": 0.17,
          "q": 0.5 
        }
      }
    }
  }
}
Show response parameter details Hide response parameter details
Response body

statusstring

Indicates the status of the job ("done", "pending", or "error")

facesmap

Key is a cluster id and value is a map representing the cluster

Show item members

List faces

Request example
CURL -H "Key: YOUR_KEY" \
https://api.muse.ai/faces/db
Response example
{
  "detected":
    {"815b4f79c996408abac72ca6866da1d7":
      {"label": null,
       "faces":
        {"fdeed8e00e5d651f737e90f343e175080244c040e0f1518b30654baebf3bef3a":
          {"moment": {
            "f": "4c2ec982c22bbb23fdc734762d7310230e09bec386fbac7946a5a6453eb16e0b",
            "s": 174.0,
            "e": 180.92,
            "x": 0.28,
            "y": 0.14,
            "w": 0.21,
            "h": 0.49,
            "q": 0.5}
          }
        }
      }
    },
  "deleted":
    {"142abde00e5d651f737e90f343e175080244c040e0f1518b30654baebf3bef3a":
      {"moment": {
        "f": "4c2ec982c22bbb23fdc734762d7310230e09bec386fbac7946a5a6453eb16e0b",
        "s": 108.0,
        "e": 108.75,
        "x": 0.74,
        "y": 0.23,
        "w": 0.13,
        "h": 0.26,
        "q": 1}
      }
    },
  "ignored":
    {"504afa20acda45ec4d690dc7aa60f612e90b0954b82a3eac369964a31d62ed4d":
      {"moment": {
        "f": "4c2ec982c22bbb23fdc734762d7310230e09bec386fbac7946a5a6453eb16e0b",
        "s": 76.0,
        "e": 77.58,
        "x": 0.19,
        "y": 0.4,
        "w": 0.08,
        "h": 0.17,
        "q": 1}
      }
    }
}
Show response parameter details Hide response parameter details
Response body

detectedmap

List of detected faces, organized into clusters, where the key is a cluster id and value is a map representing the cluster

Show item members

ignoredmap

List of faces which will not be detected in future videos, unlabeled and not organized into clusters

deletedmap

List of faces which have been deleted from the list of detected faces, unlabeled and not organized into clusters

Modify faces

Request example
curl -X POST \
        -H "Key: YOUR_KEY" \
        -H "Content-Type: application/json" \
        -d '{"0c3e64b7c7fa9753b589d90a1e0c821fa76c949256614d31bb0cd3ce6346d1c1":
            {"label": "Jenny",
            "delete": False,
            "ignore": False,
            "purge": False}}' \
        https://api.muse.ai/faces/modify
Show request parameter details Hide request parameter details
Request body

json*data

an object containing face IDs and corresponding changes

Show item members

* required

Response example
{
  "status": "ok"
}
Show response parameter details Hide response parameter details
Response body

statusstring

Indicates the success of the job (expected result: "ok")

Objects

Recognize common objects in images.

Identify objects in an image

Request example
curl -X POST \
     -H "Key: YOUR_KEY" \
     -F "file=@input.png" \
     https://api.muse.ai/objects

* Follows the API Flow steps

Show request parameter details Hide request parameter details
Request body

file*file

File to process

* required

Info

Supported video formats

JPG, JPEG, PNG, GIF, BMP, WEBP

Max file size

5GB

Response example
{
  "status": "done",
  "objects": {
    "person": [{"x": 0.508, "y": 0.281, "w": 0.514, "h": 0.233, "q": 0.5},
               {"x": 0.425, "y": 0.102, "w": 0.310, "h": 0.410, "q": 0.5}],
    "bicycle": [{"x": 0.488, "y": 0.402, "w": 0.491, "h": 0.221, "q": 0.5}]
  }
}
Show response parameter details Hide response parameter details
Response body

statusstring

Indicates the status of the job ("done", "pending", or "error")

objectsmap

Key is detected objects and value is their coordinates

Show item members

Coordinate representation

Actions

Recognize actions such as dancing, flying, driving etc. in a video.

Recognize actions

Request example
curl -X POST \
     -H "Key: YOUR_KEY" \
     -F "file=@input.mp4" \
     https://api.muse.ai/actions

* Follows the API Flow steps

Show request parameter details Hide request parameter details
Request body

file*file

File to process

* required

Info

Supported video formats

AVI, MP4, MKV, MOV, WEBM

Max file size

5GB

Max duration

1 hour

Response example
{
  "status": done,
  "actions": {
    "driving": [{"s": 5.62, "e": 6.28, "q": 0.5}, {"s": 14.83, "e": 15.46, "q": 0.5}],
    "running": [{"s": 82.29, "e": 82.93, "q": 0.5}],
    "riding": [{"s": 107.8, "e": 108.44, "q": 0.5}]
  }
}
Show response parameter details Hide response parameter details
Response body

statusstring

Indicates the status of the job ("done", "pending", or "error")

actionsmap

Key is detected action and value is a list with its occurrences

Show item members

Text

Find and recognize text in an image.

Identify text in an image

Request example
curl -X POST \
     -H "Key: YOUR_KEY" \
     -F "file=@input.png" \
     https://api.muse.ai/text

* Follows the API Flow steps

Show request parameter details Hide request parameter details
Request body

file*file

File to process

* required

Info

Supported formats

JPEG, PNG, BMP, WEBP

Max file size

50MB

Response example
{
  "status": "done",
  "text": {
    "hello": [{"x": 0.108, "y": 0.281, "w": 0.314, "h": 0.233, "q": 0.5}],
    "world": [{"x": 0.452, "y": 0.280, "w": 0.295, "h": 0.221, "q": 0.5}]
  }
}
Show response parameter details Hide response parameter details
Response body

statusstring

Indicates the status of the job ("done", "pending", or "error")

textmap

Key is a string with the text and the value is the bounding box

Show item members

Coordinate representation

Scenes

Perform splitting of videos based on shot boundaries.

Detect and cluster scenes

Request example
curl -X POST \
     -H "Key: YOUR_KEY" \
     -F "file=@input.mp4" \
     https://api.muse.ai/scenes

* Follows the API Flow steps

Show request parameter details Hide request parameter details
Request body

file*file

File to process

* required

Info

Supported video formats

AVI, MP4, MKV, MOV, WEBM

Max file size

5GB

Max duration

5 hours

Response example
{
  "status": "done",
  "scenes": [[[0.0, 8.25]], [[8.28, 16.41], [16.45, 18.92]],
             [[18.77, 25.22]], [[25.26, 45.30]]]
}
Show response parameter details Hide response parameter details
Response body

statusstring

Indicates the status of the job ("done", "pending", or "error")

scenesarray

List of scene clusters containing one or more scenes, represented by [start, end] times in seconds

Colors

Identify colors in images.

Identify colors

Request example
curl -X POST \
     -H "Key: YOUR_KEY" \
     -F "file=@input.png" \
     https://api.muse.ai/colors

* Follows the API Flow steps

Show request parameter details Hide request parameter details
Request body

file*file

File to process

* required

Info

Supported video formats

JPEG, PNG, BMP, WEBP

Max file size

50MB

Response example
{
  "status": "done",
  "colors": {"black": 0.252, "blue": 0.035, "brown": 0.08, "green": 0.01, "grey": 0.591, "orange": 0.01, "pink": 0.01, "purple": 0.0, "red": 0.0, "white": 0.01, "yellow": 0.0}
}
Show response parameter details Hide response parameter details
Response body

statusstring

Indicates the status of the job ("done", "pending", or "error")

colorsmap

Key is a string with the color and the value is the proportion of that color in the image

Sense

Detect how positive/negative the text is.

Sentiment analysis

Request example
curl -X POST \
     -H "Key: YOUR_KEY" \
     -d "this is cool and awesome and not bad" \
     https://api.muse.ai/sense
Show request parameter details Hide request parameter details
Request bodytext/plain

text*string

Text to analyze

* required

Info

Max file size

5MB

Response example
{
  "positive": 8,
  "negative": 0,
  "total": 8
}
Show response parameter details Hide response parameter details
Response body

positiveint

Positivity estimate for the text.

negativeint

Negativity estimate for the text.

totalint

Word count in the text.