Flows- A Dungeon Example

Andrew Rapo
December 8, 2016


As seen in previous posts and example, flows provide a visual mechanism for describing the flow-of-control for a skill. The SDK’s Flow Editor is used to create .flow files, which are converted (compiled) to JavaScript during the skill’s build process. Flows are especially useful for describing the complex logic required by Jibo’s Voice User Interface (VUI). In particular, flows make it easy to work with MIMs (multi-modal interaction modules) which were introduced in an earlier post.

This post explores some of the best practices for using flows - in the form of a retro (late 70’s), text-based (now VUI-based) dungeon game.

Instantiating Flows
When using flows, there is typically a main.flow that is instantiated when the skill is first opened. The main.flow file is required like this:


Then the skill’s open() method often includes code that looks like this:


The ‘blackboard’ is an object that is instantiated in the skill’s main class and then passed to the flow. It can be referenced from within any flow or subflow. In this case, it contains a reference to this skill instance, a reference to the game’s gameStateManager instance, and a reference to a class - Command -  that will be used in the flows. To instantiate the flow, the blackboard is included in an ‘options’ object and passed along with the flow object, mainFlow, to jibo.flow.run(). The flow is then called (executed) once per frame by the Jibo runtime’s main loop.

The Main Flow
When opened in the Flow Editor, the main.flow for this DungeonExample looks like this:


Begin and Flow.End
The palette on the left includes the activities that can be added to flows, including the Begin and Flow.End activities. Execution starts at the Begin activity and ends at Flow.End. When Flow.End is reached, the callback given to jibo.flow.run() is called and the flow is cleaned up.

Flow.Eval (Init)
Flow.Eval is a activity that can contain arbitrary JavaScript. When the Flow.Eval activity is selected, a panel to the right becomes active and provides a simple JavaScript editor. 

Note: It is best to limit the amount of JavaScript added to flow activities and instead invoke classes that are authored in the main Atom editor.

In this example, the Init Flow.Eval contains some JavaScript that invokes the skill’s log.info() method and accesses its gameStateManager instance.


Dungeon Welcome: Mim.Announcement
The Mim.Announcement activity - Dungeon Welcome - then delivers a TTS announcement:


The announcement prompts are defined in the mims/en-us/dungeon/DungeonWelcome.mim file. 
Note: Two data properties are passed to the mim, IDedCrewMember and sessionCount. These properties are used to choose which prompt to use and to customize the message:


The DungeonWelcome.mim file defines two prompts with different conditions. The Entry-1 prompt is used if the sessionCount == 0, if the user is playing for the first time. Otherwise the Entry-1a prompt is used. In both cases, the prompt makes use of the IDedCrewMember property to welcome the user by name.

NextCommand: Flow.Eval
After the welcome announcement, the NextCommand (Flow.Eval) activity checks to see if there are any pending commands to execute. Commands are instantiated as the result of VUI interactions. They can also be instantiated programmatically to restore the state of the game if the player is returning. NextCommand acts as a router, using the properties of the Command instance to determine which action (subflow) to trigger next. If there is no pending command, the MainMenu is triggered so the user can provide VUI-generated commands.


Command Methods
The DungeonExample class includes methods to manage pending Commands:


The Command Class
The simple Command class used by DungeonExample has a to property which is used by the NextCommand activity to route control to the appropriate subflow. Routing is accomplished by setting the transition of the NextCommand activity to the name of the command’s action - its to property. The value (transition) returned by NextCommand determines which outbound link to activate. For example, if the pending command is the result of the user saying, “Save the game," the outbound transition will be “save," that link will be activated, and the save subflow will be invoked.


Main Menu: MIM.Question
The MainMenu MIM prompts the user with the question: “What would you like to do?” As specified in the DungeonMainMenu.rule (included below), acceptable responses include:

  • Start the game
  • Buy some gear
  • Show my gear
  • Save the game
  • Quit the game

When the MainMenu MIM gets a successful parse (onSuccess) the result (results,firstGrammarTag) is stored in notepad.menuSelection so that the next activity, PushCommand can access it.

Note: The notepad is like the blackboard but its scope is limited to the current flow. Every flow has its own notepad which can be used to share data between activities.


PushCommand: Flow.Eval
The PushCommand activity uses notepad.menuSelection to instantiate a new Command.

Note: The Command class definition is passed to the flow via the blackboard and is accessed as blackboard.Command.

Then the new command is pushed (actually unshift-ed) onto the pendingCommands array.

Main Flow Recap
At this point, the control returns to the NextCommand activity and the newly-created VUI-generated command is parsed. What happens next depends on what the user said.



Save: Flow.Subflow
Flows can be treated as subflows and invoked using the Flow.Subflow activity. This allows flows to be used as modular functions. The save functionality is implemented in this way. The contents of save.flow file look like this:


The Save activity (Flow.Eval-Async) makes a call (via blackboard) to the gameStateManager’s savePlayerGameState() method. This is an asynchronous call, thus the use of the Flow.Eval-Async activity.

Quit: Flow.Subflow
The quit functionality is also implemented as a subflow - and nested inside it is another instance of save.flow. Quit contains two MIMs to as the important questions, “Are you sure you want to quit?” and if so, “Do you want to save the game?” If the user changes his/her mind and decides not to quit after all, the Quit activity returns a “cancel” transition. If the player does want to quit and does want to save, the save.flow subflow is invoked.

Finally, a decision to quit causes a “~Quit” exception to be thrown using the Flow.Throw activity. This exception will be caught by a Flow.Catch[~Quit] activity anywhere in the current flow or in a parent flow, bubbling all the way up to the main.flow if not caught earlier. In the DungeonExample, there is a Flow.Catch[~Quit] activity in the main.flow (see above). This ensures that any “~Quit” exception thrown in the game will be caught and redirected to the DungeonGoodBye announcement MIM.


Dungeon: Flow.Subflow
Once the player asks to “enter the dungeon” or “start the game,” control is handed to the dungeon.flow subflow. It looks a lot like main.flow but with a focus on in-game VUI interactions.

Note: To enable the player to quit (and save) from within the dungeon, the quit.flow subflow is re-used.

Note: If the player quits from within the dungeon, the save state will include one pending command to re-enter the dungeon. Then when the player returns, the pendingCommands array will have this ‘start’ command added to it, which will execute immediately. (See the Init activity in main.flow above.) The result is that the game will skip the MainMenu and resume in the dungeon.flow with the DungeonMenu active.


Store: Flow.Subflow
The store subflow also has a similar structure to main.flow and dungeon.flow. As seen above, the store subflow can be accessed from both the main menu and from within the dungeon. However, the DungeonMenu’s rule file (DungeonAction.rule below) does not include a rule for ‘store’. Instead, the store Command is generated programmatically when the player finds the location of the store in the dungeon. The code for this looks like:


By pushing the ‘store’ command onto the pendingActions array, it automatically gets processed just like a VUI-generated command.


Playing the Game
When a new player first enters the game, they are welcomed by name and then presented with the main menu prompt: “What would you like to do?” Any unrecognized utterance will produce a more helpful prompt, like: “You can say: buy some gear, show my gear, enter the dungeon, save the game or quit the game.” The smartest thing to do first is to buy some gear.


Once the player has the right gear, it is time to “enter the dungeon.”


After moving north, the player gets an indication that something interesting might be nearby. For instance, moving east reveals a secret door (good guess).


The Map
The game currently includes a retro, teletype-inspired stats and map screen which is continually updated. ‘P’ represents the player - standing to the west of the newly opened door.


Quit and Save
Whew. Better quit while still alive. Saying quit invokes the quit/save flow described above. In this case the game is saved along with the state information (pending command) that will restart the game in the dungeon when the player returns.


Recap: Dungeon Example
As a game, the DungeonExample skill is just a starting point, but it does serve as a good reference for making the most of the Flow Editor and flows. The patterns used in this example can also be used in a wide range of skills - to manage complex state and provide a mechanism that allows state to be saved and restored. And if anyone feels compelled to add to the game, contributions will be welcome.

Appendix - NLU Rules


TopRule = ($* $CONTROL $*) {selection = CONTROL._selection};

                $START {_selection=START._selection} |
                $STORE {_selection=STORE._selection} |
                $INVENTORY {_selection=INVENTORY._selection} |
                $SAVE {_selection=SAVE._selection} |
                $QUIT {_selection=QUIT._selection}

START = (start | (get going) | (let\'s go) | (enter $* (maze | dungeon)))

STORE = (   store | ((go | visit) $* (shopping))  |
            ((get | need | buy) $* (stuff | things | gear | equipment))

INVENTORY = ((inventory) | ((show | see | list) $* (stuff | things | gear | equipment)))

SAVE = (((save) $* ?(game)))

QUIT = (((quit | exit) $* ?(game)))


TopRule = $* (
    $MOVE {%action='move'%} |
    $OPEN {%action='open'%} |
    $SEARCH {%action='search'%} |
    $EQUIP {%action='equip'%} |
    $ATTACK {%action='attack'%} |
    $LOOK {%action='look'%} |
    $SAVE {%action='save'%} |
    $CAST {%action='cast'%} |
    $PASS {%action='pass'%} |
    $MAP {%action='map'%} |
    $EXIT {%action='exit'%} |
    $QUIT {%action='quit'%} |
    $POWERUP {%action='powerup'%} |
    $DEBUG {%action='debug'%} |
    $INVENTORY {%action='inventory'%}
) ?(at) ?( with | using) ?(a | an | the | my) $DETAIL ?($DIRECTION) ?($COUNT) ?(spaces | tiles) ?($SPELL {%spell='spell'%});

MOVE = move;
OPEN = open;
SEARCH = search;
EQUIP = (equip | use);
ATTACK = (attack | fight | strike | swing);
LOOK = look;
SAVE = save;
CAST = (run | execute| cast);
PASS = (pass | skip);
MAP = map;
EXIT = exit;
QUIT = quit;
POWERUP = powerup;
DEBUG = stats | debug;

DETAIL @= ($*){%detail = this._parsed%};

SPELL = (spell | program);

INVENTORY =((show | see | list) $* (stuff | things | gear | equipment | inventory));

DIRECTION @= (north | south | east | west){%direction = this._parsed%};
COUNT @= (one | two | three | four | five | six | seven | eight | nine | ten){%count=this._parsed%};


TopRule = $* (
    $BUY {%action='buy'%} |
    $SELL {%action='sell'%} |
    $CATALOG {%action='catalog'%} |
    $EXIT {%action='exit'%} |
    $SAVE {%action='save'%} |
    $QUIT {%action='quit'%}
) ?($COUNT) ?(a | an | the | my | another | more) $ITEM;

BUY = (buy | get | purchase);
SELL = (sell | give);
CATALOG = (catalog | (what do you have) | ((what | what\'s) $* for sale));
EXIT = (exit | done);
SAVE = save;
QUIT = (quit);

COUNT @= (one | two | three | four | five | six){%count=this._parsed%};

ITEM @= ($*){%item = this._parsed%};


Andrew Rapo
Executive Producer, Business Development & Marketing