Inter-CPU Channels – Oberon RTS

Overview

With two CPUs in the FPGA, a means of communication between the processors, and the processes running thereon, is required. This page describes a basic two-way buffered channel, implemented as circular buffers (FIFOs).

Structure

In each direction, ie. from CPU 0 to CPU 1, and vice versa, we have two synchronised FIFOs:

32 bit data
16 bit control data

The control data is experimental. Each (32 bit) data item can be tagged with a control data item, which might help to simplify the communication protocol, as control and status information does not have to be sent using the data channel, which invariably would require a state machine (or equivalent) to receive and decode a non-trivial, multi-part message.

Driver API

The driver API is very simple, just providing the raw means to send and receive data. Here’s the version that uses device signalling in case the send buffer is full, or the receive buffer empty. No timeout is used, to keep the API simple, and also because the channel is an internal device without potential connection issues, and not scheduled, or stuck, processes on either end should be detected by other means anyway.

MODULE CPUCp;

  IMPORT SYSTEM, DevSignals, Oberon;
  
  CONST
    (* addresses *)
    StatusAdr = -188;
    CtrlAdr   = -192;
    DataAdr   = -196;
    
    (* status bits *)
    SendFull = 1;
    RecEmpty = 4;
    
    (* ... *)
    
    (* device signals as wired *)
    DevSigSendNotFull = DevSignals.CPUCsendNotFull;
    DevSigRecNotEmpty = DevSignals.CPUCrecNotEmpty;

  PROCEDURE Send*(data, ctrl: INTEGER);
    VAR res: INTEGER; (* dummy: no timeout used *)
  BEGIN
    IF SYSTEM.BIT(StatusAdr, SendFull) THEN
      Oberon.AwaitDevSignal(DevSigSendNotFull, 0, res)
    END;
    SYSTEM.PUT(CtrlAdr, ctrl); (* write ctrl first *) 
    SYSTEM.PUT(DataAdr, data)
  END Send;
  
  PROCEDURE Receive*(VAR data, ctrl: INTEGER);
    VAR res: INTEGER; (* dummy: no timeout used *)
  BEGIN
    IF SYSTEM.BIT(StatusAdr, RecEmpty) THEN
      Oberon.AwaitDevSignal(DevSigRecNotEmpty, 0, res)
    END;
    SYSTEM.GET(CtrlAdr, ctrl); (* read ctrl first *) 
    SYSTEM.GET(DataAdr, data)
  END Receive;

END CPUCp.

A version not using device signalling is available as well, with which a program would first check if the send buffer is not full, or the receive buffer not empty, and only then proceed to write to, or read from, the channel.

The wiring in the FPGA takes care of “crossing the wires” – ie. the CPUs writing to one hardware device, but reading from the other, and reading the corresponding status bits – to ensure the same IO addresses and bit indices for both CPUs.

Message Protocols

The driver API provides the basics to implement application specific message protocols.

I am still mulling over a basic system level protocol, eg. to execute commands on CPU 1, ie. the one not connected to the Astrobe terminal.

Alternative Driver API

The use of control data is not mandatory, from the hardware point of view. Also, the control data does not need to be written with every 32 bit data item: if the control data is the same for, say, a series of data items, it only needs to be written once, and subsequent data items will use the same control data until changed, as the control data is written to an input register first, and only transferred to the FIFO when the 32 bit data is written.

The driver provides procedures for this more direct use as well.

MODULE CPUCp;

  (* ... *)

  PROCEDURE PutCtrl*(ctrl: INTEGER);
  BEGIN
    SYSTEM.PUT(CtrlAdr, ctrl)
  END PutCtrl;
  
  PROCEDURE PutData*(data: INTEGER);
    VAR res: INTEGER; (* dummy: no timeout used *)
  BEGIN
    IF SYSTEM.BIT(StatusAdr, SendFull) THEN
      Oberon.AwaitDevSignal(DevSigSendNotFull, 0, res)
    END;
    SYSTEM.PUT(DataAdr, data)
  END PutData;

  (* ... *)

END CPUCp.

The corresponding reading procedures are omitted in the listing above. When using this part of the API – which of course can be mixed with the Send and Receive API presented before –, it’s important to write and read the control data before the 32 bit data. Only the 32 bit data write and read operations make changes to the FIFOs, hence not reading control data will not get the FIFOs out of sync.

Note that this also allows a process to read the control data to check if the message in the buffer is addressed to it, and skip reading the data if not, leaving the channel buffers unchanged, to allow other processes to do the same. This way, a direct process-to-process communication can be implemented, without an additional mechanism to distribute the incoming messages to the recipient processes, provided the overall set-up and timing allows for that simple scheme.

Concurrent Access

With processes running on two CPUs, two processes might attempt to write and read from the channel at the same time, ie. in exactly the same clock cycle. It’s not possible to synchronise these concurrent accesses, as without the channel we don’t even have the possibility to exchange any synchronisation signals. The FIFO’s are dual port types, and the implementation can cope with write and read access in the same clock cycle (I hope).

Inter-Process Communication?

The inter-CPU channels might also provide a useful concept for inter-process communication on the same CPU, without any need for synchronisation via semaphores or similar means. For this, a set of available channels would need to be instantiated, of which both send and receive are accessed from the same CPU. To be explored.