Accomplishments
This week I accomplished one of the hard parts of the FPU job manager: writing the FSM for the Convolutional Forward module.
It’s worth mentioning that there were easier options for implementing Convolutional Forward than to define the operation in an FSM. We could have chosen to write convolutional forward in C, compile it to RISCV-assembly, and copy the code to our FPGA and use that, but this would require precious on-board memory that we need to store input samples. Writing Convolutional forward as an FSM is the fastest and smallest solution to our problem, and doing so will maximize our model throughput. Convolutional backward (gradient with respect to the inputs, filters, or bias) are essentially the same nested loops above, so implementing those operations will be significantly easier if I use the code above as a starting point.
I now know the exact requirements that the FPU Job Manager needs: 32×32-bit registers that can be accessed with multiple reads and multiple writes per cycle (I will have to declare an array of registers and expose all of the wires instead of writing a Register File), and some specific ALU/FPU instructions:
- offset = channel_index + (num_channels * i) + ((num_channels * image height) * j)
- x = x + 1 (integer increment)
- x = y + z (integer addition)
- floating-point multiply
- floating-point addition
Like I mentioned last week, having Convolutional Forward completely defined will cause a lot of other pieces to fall into place. Next week, I plan on implementing some of these ALU/FPU and register requirements in SystemVerilog.
Here’s what my definition looks like:
pseudocode:
w_o_fac <- x.height * x.channels
j <- 0
j_x <- (-1 * pad)
while(j < z.width):
w_o <- w_o_fac * j_x
i <- 0
i_x <- -1 * pad
while(i < z.height):
beta <- 0
while(beta < f.width):
alpha <- 0
while(alpha <- f.height):
gamma <- 0
while(gamma < f.in_channels):
delta <- 0
while(delta < f.out_channels):
z[delta, i, j] <- z[delta, i, j] + (f[delta, gamma, alpha, beta] * x[gamma, i_x + alpha, j_x + beta])
delta <- delta + 1
gamma <- gamma + 1
alpha <- alpha + 1
beta <- beta + 1
delta <- 0
while(delta <- z.channels):
z[delta, i, j] <- z[delta, i, j] + b[delta]
delta <- delta + 1
i <- i + 1
i_x <- i_x + stride
j <- j + 1
j_x <- j_x + stride
State transitions:
// STATE == START_LOAD1
r1 <- stride
r5 <- filter.output_channels
r9 <- x.height
nextState = START_LOAD2
// STATE == START_LOAD2
r2 <- pad
r6 <- filter.input_channels
r10 <- x.width
nextState = START_LOAD3
// STATE == START_LOAD3
r3 <- output_height
r7 <- filter.height
nextState = START_LOAD4
// STATE == START_LOAD4
r4 <- output_width
r8 <- filter.width
*z <- output_height
z++
nextState = START_CALC
// STATE == START_CALC
*z <- output_width
z++
r11 <- r9 * r6 // w_o_factor = x.height * x.channels
r12 <- ~r2 + 1 // j_x = -1 * pad
r19 <- 0 // j = 0
nextState = J_LOOP
// STATE == J_LOOP
r13 <- r11 * r12
if(r19 == r4): // if j == z.width
nextState = DONE
else:
r21 <- 0 // i = 0
r20 <- ~r2 + 1 // i_x = -1 * pad
nextState = I_LOOP
// STATE == I_LOOP
if(r21 == r3): // if j == z.height
r19 <- r19 + 1 // j += 1
r12 <- r12 + r1 // j_x += stride
nextState = J_LOOP
else:
r22 <- 0 // beta = 0
r28 <- r20 * r6 // r28 = f.input_channels * i_x
nextState = BETA_LOOP
// STATE == BETA_LOOP
if(r22 == r8): // if beta == f.width
r25 <- 0 // delta = 0
z <- z - r5 // Reset z counter, we’re going to iterate over channels again
b <- 2
nextState = BIAS_LOOP_LOAD
else:
r23 <- 0 // alpha = 0
nextState = ALPHA_LOOP
// STATE == ALPHA_LOOP
if(r23 == r7): // if alpha == f.height
r22 <- r22 + 1
nextState = BETA_LOOP
else:
r24 <- 0 // gamma = 0
nextState = GAMMA_LOOP
// STATE == GAMMA_LOOP
if(r24 == r6): // if gamma == f.input_channels
r23 <- r23 + 1 // alpha = 0
nextState = ALPHA_LOOP
else:
r25 <- 0 // delta = 0
r27 <- r24 + (r6 * (r20 + r23)) + (r11 * (r12 + r22)) // x offset
nextState = DELTA_LOOP_LOAD
// STATE == DELTA_LOOP_LOAD
r14 <- *z // r14 <- z[delta, i, j]
r15 <- *f // r15 <- f[delta, gamma, alpha, beta]
r16 <- *r27 // r16 <- x[gamma, i_x + alpha, j_x + beta]
if(mem(z).done && mem(f).done && mem(x).done):
nextState = DELTA_LOOP_CALC1
else:
nextState = DELTA_LOOP_LOAD
// STATE == DELTA_LOOP_CALC1
r17 <- r15 * r16 // r17 <- f[delta, gamma, alpha, beta] * x[gamma, i_x + alpha, j_x + beta]
nextState = DELTA_LOOP_CALC2
// STATE == DELTA_LOOP_CALC2
r18 <- r14 + r17 // r18 <- z[delta, i, j] + (f[delta, gamma, alpha, beta] * x[gamma, i_x + alpha, j_x + beta])
nextState = DELTA_LOOP_STORE
// STATE == DELTA_LOOP_STORE
*z <- r18
if(mem(z).done && r25 == r5): // if(memory is done writing and delta == f.output_channels)
r24 <- r24 + 1
nextState = GAMMA_LOOP
else if(mem.done):
nextState = DELTA_LOOP_STORE
else:
r25 <- r25 + 1 // delta += 1
z++
f++
nextState = DELTA_LOOP_LOAD
// STATE == BIAS_LOOP_LOAD
r14 <- *z // r14 <- z[delta, i, j]
r15 <- *b // r15 <- b[delta]
if(mem(z).done && mem(b).done):
nextState = BIAS_LOOP_CALCS
else:
nextState = BIAS_LOOP_LOAD
// STATE == BIAS_LOOP_CALCS
r16 <- r14 + r15
nextState = BIAS_LOOP_STORE
// STATE == BIAS_LOOP_STORE
*z <- r16
if(mem(z).done && r25 == r5):
z++
b++
r21 <- r21 + 1 // i += 1
r20 <- r20 + r1 // i_x += stride
nextState = I_LOOP
else if(mem(z).done):
r25 <- r25 + 1
nextState = BIAS_LOOP_LOAD
else:
nextState = BIAS_LOOP_STORE
// STATE == DONE
Schedule
I remain on the schedule that I proposed last week.
Accomplishments for Next Week
Next week will be time to start implementing the FPU Job Manager. Now that I know the upper limit for the resources that the FPU Job Manager needs, I can be confident that I won’t have to redesign it. Although I only have a couple of FSM controllers defined, I want to go ahead with the implementation so that I can solve any unexpected problems related to SystemVerilog implementation of the modules and memory accesses.