SEI Insertion

OvenMediaEngine inserts Supplemental Enhancement Information (SEI) into live streams to deliver custom data with the video at frame-level precision.

Overview

Two types of SEI can be inserted. Which one you use depends on the data you need to deliver.

SEI type	Configured through	Description
UserDataUnregistered (type 5)	Send Event API XML configuration	Delivers data you define. The payload follows a predefined format -- UUID, Timestamp, then your data -- which OvenPlayer parses for you.
PictureTiming (type 1)	XML configuration	Delivers a SMPTE timecode in the field the H.264 specification defines for it. H.264 only.

info

PictureTiming is configured only in the EventGenerator XML. The Send Event API rejects it with 400 Bad Request.

Inserting SEI UserDataUnregistered

UserDataUnregistered (SEI type 5) carries data you define. The payload is wrapped in a predefined format -- UUID, Timestamp, then your data -- so a consumer such as OvenPlayer can identify the message and parse it. See Predefined Payload Format.

Send one message at a time with the Send Event API, or repeatedly from the EventGenerator XML.

Using the Send Event API

Use the SendEvent REST API to insert an SEI message into a running stream.

Request

POST /v1/vhosts{vhost}/apps/{app}/streams/{stream}:sendEvent

Header

Authorization: Basic {credentials}

# Authorization
Credentials for HTTP Basic Authentication created with <AccessToken>

Body

{
    "eventFormat": "sei",
    "eventType": "video",
    "events": [
        {
            "seiType": "UserDataUnregistered",
            "data": "OvenMediaEngine"
        }
    ]
}

POST /v1/vhosts{vhost}/apps/{app}/streams/{stream}:sendEvents

Header

Authorization: Basic {credentials}

# Authorization
Credentials for HTTP Basic Authentication created with <AccessToken>

Body

[
  {
      "eventFormat": "sei",
      "eventType": "video",
      "events": [
          {
              "seiType": "UserDataUnregistered",
              "data": "OvenMediaEngine"
          }
      ]
  }
]

Parameter	Required	Description
`eventFormat`	Y	Specifies the event format (use `sei` format).
`eventType`	N	Specifies the event type. Default: `video`
`events`	Y	Contains event data values.
`event.seiType`	N	Specifies SEI type. Only `UserDataUnregistered` is accepted here. Default: `UserDataUnregistered`
`event.data`	Y	The data to transmit. Supports the `${EpochTime}` macro, replaced with the server's epoch time in milliseconds at insertion (e.g., 1747147513056).

Using the ${EpochTime} Macro

${EpochTime} in event.data is expanded when the SEI enters the stream, not when the request is received, so the value is the insertion time.

{
    "eventFormat": "sei",
    "eventType": "video",
    "events": [
        {
            "seiType": "UserDataUnregistered",
            "data": "Hi! OvenMediaEngine! CurrentTime:${EpochTime}"
        }
    ]
}

Response

200 Ok

Header

Content-Type: application/json

Body

{
    "message": "OK",
    "statusCode": 200
}

400 Bad Request

Header

Content-Type: application/json

Body

{
    "message": "eventFormat(string) and events(array) are required",
    "statusCode": 400
}

{
    "message": "eventFormat is not supported: [XXX]",
    "statusCode": 400
}

{
    "message": "Could not make events data",
    "statusCode": 400
}

{
    "message": "eventType must be string",
    "statusCode": 400
}

{
    "message": "eventType is not supported: [XXX]",
    "statusCode": 400
}

{
    "message": "Unknown seiType or Invalid format",
    "statusCode": 400
}

500 Internal Server Error

Header

Content-Type: application/json

Body

{
    "message": "Could not inject event: [XXX]",
    "statusCode": 500
}

Using XML Configuration

For continuous insertion, define the events in an XML file and enable EventGenerator in Server.xml.

Server.xml: Enable EventGenerator by adding <Application><EventGenerator>.

<?xml version="1.0" encoding="UTF-8"?>
<Server version="8">
  ...
  <VirtualHosts>
    <VirtualHost>
      <Applications>
        <Application>
          ...
          <EventGenerator>
            <Enable>true</Enable>
            <Path>events/send_event_info.xml</Path>
          </EventGenerator>
        </Application>
      </Applications>
    </VirtualHost>
  </VirtualHosts>
</Server>

Element Required Description

<Enable>

Sets activation status to true or false.

Default: false

<Path> Y Sets the path to the XML file defining SEI insertion details. If a relative path is specified, the directory containing the Server.xml file is used as the base.

Event definition XML: Create the file at the path given in Server.xml. In this example, send_event_info.xml.

<?xml version="1.0" encoding="UTF-8"?>
<EventInfo>
  <Event>
    <Enable>true</Enable>
    <SourceStreamName>stream*</SourceStreamName>
    <Interval>2000</Interval>
    <EventFormat>sei</EventFormat>
    <EventType>video</EventType>
    <Values>
      <SeiType>UserDataUnregistered</SeiType>
      <Data>Hi! OvenMediaEngine! CurrentTime:${EpochTime}</Data>
      <KeyframeOnly>true</KeyframeOnly>
    </Values>
  </Event>
</EventInfo>

Parameter	Required	Description
`<Enable>`	Y	Sets activation status to `true` or `false`. Default: `false`
`<SourceStreamName>`	Y	Specifies the target stream name. Supports wildcards (`*`).
`<Interval>`	N	Sets event occurrence interval in milliseconds (ms). Omit it (or set 0) to send the event once per stream, at the start.
`<EventFormat>`	Y	Specifies event format (use `sei` format).
`<EventType>`	N	Specifies the event type. Default: `video`
`<Values>`	Y	Contains the value of the event data.
`<Values><SeiType>`	N	Specifies SEI type. Default: `UserDataUnregistered`
`<Values><Data>`	Y	Specifies custom data to be inserted into SEI. Supports the `${EpochTime}` macro, replaced with the server's epoch time in milliseconds at insertion (e.g., 1747147513056).
`<Values><KeyframeOnly>`	N	Specifies the target for event insertion. If set to `true`, the event will be inserted into the first keyframe after the specified interval has passed (Supported from version 0.18.2.0+). Default: `false`

info

Changes to the event definition XML file are applied without a restart.

Predefined Payload Format

The payload always carries UUID and Timestamp values ahead of your data:

0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| uuid_iso_iec_11578(128)                                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Timestamp (64)                                                |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data (Payload Size - UUID(128) - Timestamp(64))               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ...                                                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Field	Size (bits)	Description
UUID	128	Marks the payload as following this format. Always set to `464d4c47-5241-494e-434f-4c4f-55524201`
Timestamp	64	Epoch time in milliseconds
Data	Varies depending on custom data	Custom data

Inserting SEI PictureTiming

PictureTiming (SEI type 1) writes a SMPTE timecode into the field the H.264 specification reserves for it, so consumers read hours/minutes/seconds/frames from it directly.

The first stamped picture takes the time of day, and the timecode advances with the media clock from there, so it does not drift. <Timezone> chooses which clock that first reading comes from.

The timecode is written whether the stream is transcoded or passed through.

info

H.264 only. H.265 moved the clock timestamps to a separate time_code SEI (payload type 136), which is not implemented. On an H.265 track the event is ignored and a warning is logged.

Using XML Configuration

Set <SeiType> to PictureTiming in the XML file <EventGenerator><Path> points at. Enabling EventGenerator in Server.xml is the same as for UserDataUnregistered above.

<?xml version="1.0" encoding="UTF-8"?>
<EventInfo>
  <Event>
    <Enable>true</Enable>
    <SourceStreamName>stream*</SourceStreamName>
    <EventFormat>sei</EventFormat>
    <EventType>video</EventType>
    <Values>
      <SeiType>PictureTiming</SeiType>
      <Timezone>UTC</Timezone>
    </Values>
  </Event>
</EventInfo>

Parameter	Required	Description
`<Values><SeiType>`	Y	Set to `PictureTiming`.
`<Values><Timezone>`	N	Which clock the timecode starts from. `UTC` (the default), `Local`, or a fixed offset written `+09:00`, `-0500` or `+09`.

info

Leave <Interval> out. Every picture is stamped, so there is no insertion period to set, and omitting it sends the event once per stream -- all a timecode needs.

Stamping then continues for the lifetime of the stream. Removing the event or setting <Enable>false</Enable> does not affect a stream that is already running; restart it.

Choosing the Timezone

<Timezone> is read once, for the first picture of the stream. Later timecodes come off the media clock, so the zone only moves the starting point.

Value	Meaning
`UTC`	Coordinated Universal Time. The default, and what you get by leaving the element out. `Z` means the same.
`Local`	The zone the server process runs in, DST included.
`+09:00`	A fixed offset from UTC. `+0900` and `+09` mean the same, and the widest accepted offset is `+14:00`.

UTC is the default because it does not depend on where the stream runs. Local is only as good as TZ on the process: containers often ship no zone, so it resolves to UTC anyway, and two nodes configured differently stamp the same stream differently.

The sign is required and the minutes are not, so +09:00, +0900 and +09 are the same value. +00:00 is UTC. The widest offset accepted is +14:00; a value beyond that, or one that cannot be read, is rejected when the file is loaded and the event is skipped with a warning naming the XML.

<Values>
  <SeiType>PictureTiming</SeiType>
  <Timezone>+09:00</Timezone>   <!-- Seoul, Tokyo -->
</Values>

A fixed offset does not observe daylight saving. +09:00 (Seoul, Tokyo), +05:30 (Mumbai), +05:45 (Kathmandu), -03:00 (Sao Paulo) and -05:00 (Bogota, Lima) are safe because those zones do not shift. One that does -- Berlin at +01:00 in winter and +02:00 in summer, Los Angeles at -08:00 and -07:00 -- will be an hour out for part of the year, so use Local with TZ set on the process, or edit the offset when the zone shifts.

IANA names such as Asia/Seoul are rejected: resolving one would mean changing the process-wide TZ, which is not safe while other threads are reading the clock.

The zone is per event and latched: each track reads it when its first picture arrives, so editing <Timezone> afterwards does not move a running timecode.

info

SMPTE 12M carries a time of day and no zone, so nothing in the bitstream records which one was used. A consumer that needs to know has to be told out of band.

Verifying the Timecode

ffmpeg decodes the SEI field by field, the quickest way to confirm what a consumer sees:

ffmpeg -i <stream_or_file> -c copy -bsf:v trace_headers -f null - 2>&1 | grep -A14 "Picture Timing"

Picture Timing
  pic_struct                 0000 = 0
  clock_timestamp_flag[0]       1 = 1
  ...
  n_frames               00010100 = 20
  seconds_value            111001 = 57
  minutes_value            110011 = 51
  hours_value               10011 = 19

If the Picture Timing section never appears, check pic_struct_present_flag in the SPS: when it is 0, ffmpeg does not read the timecode.

ffmpeg -i <stream_or_file> -c copy -bsf:v trace_headers -f null - 2>&1 | grep pic_struct_present_flag

OvenPlayer Integration

OvenPlayer parses the inserted SEI and passes the UUID, Timestamp, and custom data to your application.

info

SEI is delivered on WebRTC playback and requires OvenPlayer 0.10.39 or later. See the OvenPlayer Github for installation.

Because of the Web Worker CORS policy, ovenplayer.js must be self-hosted, and RTCTransformWorker.worker.worker.js must sit in the same directory as it. Both files are in the dist directory of the release.

Code Example

var player = OvenPlayer.create('player', {
  sources: [
    {
      type: 'webrtc', // Playing WebRTC stream
      file: 'wss://[YOUR_OvenMediaEngine]:3333/app/stream'
    }
  ],
  parseStream: {
    enabled: true // Enable H.264 NAL parsing
  }
});

function toAsciiString(byteArray) {
  return String.fromCharCode.apply(null, byteArray);
}

player.on('metaData', function (metadata) {
  console.log('MetaData:', metadata);
  /* Output:
    {
      type: 'sei',
      nalu: Unit8Array(33),
      sei: {
        type: 5,
        size: 39,
        payload: Unit8Array(39)
      },
      registered: true,
      uuid: '464d4c47-5241-494e-434f-4c4f-55524201',
      timecode: 1739851602778,
      userdata: Unit8Array(15)
    }
  */

  console.log(`Convert user data to string: ${toAsciiString(metadata.userdata)}`);
  /* Output:
    Convert user data to string: OvenMediaEngine
  */
});

Player Initialization:

Call OvenPlayer.create() to create a player in the specified div.
Specify the stream type and URL in the sources array. SEI is only supported in WebRTC streams.
Enable H.264 Network Abstraction Layer (NAL) parsing with parseStream.enabled: true. It is required for SEI processing.

SEI Data Processing:

Register a player.on('metaData', callback) listener to process each SEI as it arrives.
The callback parameter carries the UUID, Timestamp, and custom data.

metaData Event Callback Parameters

Field	Description
`type`	Always set to `sei`, indicating this is SEI metadata
`nalu`	Uint8Array containing raw Network Abstraction Layer Unit (NALU) data of the SEI
`sei`	SEI parsing result containing the following sub-fields: `type`: SEI type `size`: Payload size `payload`: Raw SEI payload data (Uint8Array)
`registered`	Indicates whether the SEI follows the predefined payload format. If true, the following fields are included
`uuid`	(when registered=true) Unique identifier carried in the SEI
`timecode`	(when registered=true) Timestamp (milliseconds) when the SEI was inserted
`userdata`	(when registered=true) Uint8Array containing custom data, parsed according to the application's requirements

Overview​

Inserting SEI UserDataUnregistered​

Using the Send Event API​

Using XML Configuration​

Predefined Payload Format​

Inserting SEI PictureTiming​

Using XML Configuration​

Choosing the Timezone​

Verifying the Timecode​

OvenPlayer Integration​

Code Example​

Overview

Inserting SEI UserDataUnregistered

Using the Send Event API

Using XML Configuration

Predefined Payload Format

Inserting SEI PictureTiming

Using XML Configuration

Choosing the Timezone

Verifying the Timecode

OvenPlayer Integration

Code Example