Managing Code with Custom Flow Activities

Andrew Rapo
January 20, 2017

Flow Editor Activities (Shapes)

The Flow Editor provides a number of ‘activities’ (reusable shapes) that are useful for authoring flows. The core set of activities includes Begin, Parallel Begin, Flow.end, Flow.Subflow, Flow.Eval, etc. By using these activities and referencing custom skill code, just about any feature can be implemented. However, not all of the potential uses of activities have been anticipated so the SDK provides a way to implement custom activities.

CustomShapes_PlayVideoJS_shapes.png

 

A Custom Activity Example: PlayVideoJS

One way to play a video on Jibo’s screen is by using a DOM video element. The code for creating and managing a video element can be included in the skill’s TypeScript codebase or it can be implemented using a Flow.Eval-Async. But it can also be implemented as a custom activity making it easily used in a skill’s flows. 

CustomShapes_PlayVideoJS_flow.png

Schemas

When a custom activity is added to a skill project it shows up in the activity palette along with the core set of activities. To let the skill know about the custom activity a schema describing the activity’s interface must be added to the skill’s schemas directory - in a schema json file.

 

CustomShapes_PlayVideoJS_tree.png

The schema for the PlayVideoJS (included in schemas/schema.json) activity looks like this:

CustomShapes_PlayVideoJS_schema.png

The schema defines the name of the activity, “PlayVideoJS”, indicates that it can be used in flows, that it’s color in the palette will be “JiboBlue”, and that it should be listed in position 221 (after the core activities, near the bottom of the list).

The schema also defines the fields that can be configured in the Flow Editor, including:

  • getVideoPath - a function that will be called when the activity is started through which it will get the name of the video file to play
  • playAudio - a boolean value that determines whether the audio track of the video will be played

Once the schema is defined, the activity will show up in the palette, but in order to be functional there must be a TypeScript implementation.

 

The Implementation

Because this custom activity is specific to this skill, its implementation is included in the skill’s src directory. PlayVideoJS.ts contains the implementation.

CustomShapes_PlayVideoJS_src.png

PlayVideoJS.ts - start

All custom activities extend the base class, jibo.flow.ActivityImplementation and override the following methods:

  • start() - called when the activity is started
  • update() - called each frame while the activity is running
  • stop() - called when the activity is stopped
  • destroy() - called to destroy - clean up - the activity

The start method of PlayVideoJS looks like this:

CustomShapes_PlayVideoJS_start.png

In the start() method, the options property (this.options) provides access to the configuration fields described in the schema: i.e. getVideoPath (Function) and playAudio (boolean). 

Note: assetPack field/vlaue is temporary and can be ignored. In an upcoming update it will be automatically passed to the activity. It will default to an empty string which is fine.

The getVideoPath() function is called and the returned value is used to set the videoUrl property. For this example, the video is expected to be in the skill’s video folder (in the root directory of the skill). Using jibo.utils.PathUtils.setDefaultPath('video', path) is a convenient way to get the fully qualified path to the video file. Then to start the video, the activity’s play() method is called.



PlayVideoJS.ts - play

The play() method takes care of setting up the DOM video element (via createVideoElement()). The videoElement’s muted property is set based on the playAudio value retrieved from the activity’s configuration.  The videoElement’s onended handler is set so that when the video finishes playing it will set the status of the activity to jibo.bt.Status.SUCCEEDED which lets the flow executor know that the activity is done.



CustomShapes_PlayVideoJS_play.png



PlayVideoJS.ts - createVideoElement

The createVideoElement() method manipulates the DOM by finding the ‘face’ element (Jibo’s root DOM element) and the root canvas element. If these are found successfully (and a ‘video-element’ does not yet exist - it shouldn’t at this point) a new video element is created and added before (on top of) the root canvas element. This ensures that the video will play on top of everything else. 

CustomShapes_PlayVideoJS_createVideoElement.png



PlayVideoJS.ts - stop

As mentioned above, the stop()method is called automatically when the activity’s status changes to jibo.bt.Status.SUCCEEDED. At this point, if the video has not already finished playing it will be paused.



CustomShapes_PlayVideoJS_stop.png



PlayVideoJS.ts - destroy

The destroy() method is automatically called and makes sure that all resources created by the activity are released and cleaned up.



CustomShapes_PlayVideoJS_destroy.png



Registering the Code

The final step is to register the PlayVideoJS.ts code with the flow system. A good place to do this is in the skill’s index.ts file. Like this: 

CustomShapes_PlayVideoJS_index.png

The call to register the activity, jibo.flow.register('PlayVideoJS', 'project', PlayVideoJS), links the information in the schema with the activity’s implementation. Once this is done, the new activity is ready to go.



The Activity in Action

Here is a video showing the PlayVideoJS activity in action. It plays a video called “9amClock_Ani.mp4” that shows an animation of a clock (about 4 seconds).



Custom Flow Activities

With custom Flow Activities, the tool palette for authoring flows can be extended to suit any skill and any developer.



As always let us know if you have any questions or comments in our developer forum.

Andrew Rapo
Executive Producer, Business Development & Marketing