Running flatten and opt passes after custom techmap

[asraa]

Hey everyone,

A quick question here about using custom techmap on a hierarchical model. It seems like running opt passes on a flattened module that was techmapped to custom cells doesn't work as effectively as when the module is not techmapped to the custom cells.

Background:
We'd like to remove all unused wires for a later part of our pipeline. The opt_expr; opt passes seem to do this, but only if they are performed before applying a custom techmap.

These unused wires appear after running flatten on a hierarchical module. In particular some intermediate results are preserved but never used.

However, our modules are very large (1.7M cells), and if we run flatten before the custom techmap, we reach a segfault during the custom techmap:

libc++abi: terminating due to uncaught exception of type std::length_error: hash table exceeded maximum size.
Design is likely too large for yosys to handle, if possible try not to flatten the design.

Reproducer

Take this hierarchical module with this map file:

module wirewire(input wire signed [2:0] a, output wire signed [1:0] b);
   wire [2:0] v1;

   sub sub_i(a, v1);
   assign b = v1[1:0];   
endmodule // wirewire

module sub(input wire signed [2:0] a, output wire signed [2:0] b);
   wire signed [2:0] c;

   assign c = 3'b111;
   assign b = a + c;
endmodule

Running techmap before the flatten and opt passes results in an unneeded wire \sub_i.b and unneeded assignment.

$ yosys -p 'read_verilog /tmp/test.v; hierarchy -check -top wirewire; opt; techmap; opt; abc -lut 3 -fast; opt_clean -purge; techmap -map map_lut_to_lutmux3.v; opt_clean -purge; flatten; opt_expr; opt; opt_clean -purge; clean; write_verilog -noexpr -noattr /tmp/test.netlist.v'

$ cat /tmp/test.netlist.v
/* Generated by 0.28+6 */

module wirewire(a, b);
  input [2:0] a;
  wire [2:0] a;
  output [1:0] b;
  wire [1:0] b;
  wire [2:0] \sub_i.b ;
  lut3 _0_ (
    .A(a[0]),
    .B(a[1]),
    .C(32'd0),
    .P0(1'h1),
    .P1(1'h0),
    .P2(1'h0),
    .P3(1'h1),
    .P4(32'd0),
    .P5(32'd0),
    .P6(32'd0),
    .P7(32'd0),
    .Y(b[1])
  );
  lut3 _1_ (
    .A(a[0]),
    .B(a[1]),
    .C(a[2]),
    .P0(1'h1),
    .P1(1'h0),
    .P2(1'h0),
    .P3(1'h0),
    .P4(1'h0),
    .P5(1'h1),
    .P6(1'h1),
    .P7(1'h1),
    .Y(\sub_i.b [2])
  );
  lut3 _2_ (
    .A(a[0]),
    .B(32'd0),
    .C(32'd0),
    .P0(1'h1),
    .P1(1'h0),
    .P2(32'd0),
    .P3(32'd0),
    .P4(32'd0),
    .P5(32'd0),
    .P6(32'd0),
    .P7(32'd0),
    .Y(b[0])
  );
  assign \sub_i.b [1:0] = b;
endmodule

If we run flatten and opt before the techmap step:

yosys -p 'read_verilog /tmp/test.v; hierarchy -check -top wirewire; opt; techmap; opt; abc -lut 3 -fast; opt_clean -purge; flatten; opt_expr; opt; opt_clean -purge; techmap -map map_lut_to_lutmux3.v; opt_clean -purge; clean; write_verilog -noexpr -noattr /tmp/test_before.netlist.v'

$ cat /tmp/test_before.netlist.v
/* Generated by 0.28+6 */

module wirewire(a, b);
  input [2:0] a;
  wire [2:0] a;
  output [1:0] b;
  wire [1:0] b;
  lut3 _0_ (
    .A(a[0]),
    .B(a[1]),
    .C(32'd0),
    .P0(1'h1),
    .P1(1'h0),
    .P2(1'h0),
    .P3(1'h1),
    .P4(32'd0),
    .P5(32'd0),
    .P6(32'd0),
    .P7(32'd0),
    .Y(b[1])
  );
  lut3 _1_ (
    .A(a[0]),
    .B(32'd0),
    .C(32'd0),
    .P0(1'h1),
    .P1(1'h0),
    .P2(32'd0),
    .P3(32'd0),
    .P4(32'd0),
    .P5(32'd0),
    .P6(32'd0),
    .P7(32'd0),
    .Y(b[0])
  );
endmodule

The unused wire from the submodule and assignments go away.

Questions

(0) Why does the custom techmap prevent Yosys opt steps from removing the unused wires?

(1) If I run custom techmap on the flattened module, sometimes I run into a hash table size exception: why does the custom techmap (which is just rewriting LUT1 and LUT 2 to custom LUT 3) require this?

(2) Is there any way to indicate more info in the custom techmap to allow opt passes to fully optimize the module in the way it does before flatten? Alternatively, because the map file is rewriting LUT 1 and LUT 2 both as LUT 3, is there a way to specify only LUT 3?

(3) Is there some way of running the techmap step with a configuration that would allow it to avoid the hash map exception after I run flatten? I am running opt passes before the techmap, so I'm not sure what else to do here.

[tcal-x]

Quick note -- have you tried opt -purge to get rid of the unused wires? The -purge option is important. Even when using just opt_clean.

[asraa]

Yes! I've been running opt_clean -purge; after each step - is that different opt -purge

[tcal-x]

opt_clean and clean are identical outside of verbosity, according to the online help.

[j2kun]

Is there a good reason why techmap is using up so much RAM? I thought it was doing a local transformation per cell (matching against the rules in the techmap file), and I imagine the only reason this would exhaust RAM is if it were building an index of the entire module.

[nakengelhardt]

Nearly every pass in Yosys does unfortunately need to build at least one index of the entire module (a sigmap) to do even basic operations... and a sigmap is indeed used at multiple points in techmap.

[j2kun]

What is the size of the largest hash table in this case?

[asraa]

https://github.com/YosysHQ/yosys/blob/master/kernel/hashlib.h#L195

it seems like it's up to 371064217 because i'm getting the error after this (32-bit size). I don't know how that corresponds to the # of cells though, and that's the part that I'd like to know or understand.

[tcal-x]

I am working directly with @asraa on this. One possibility is that the desired optimization isn't happening because we haven't provided a black box model for lut3 (so Yosys doesn't know which pins are inputs vs output).

[asraa]

Thank you! Yes, to summarize @tcal-x 's findings, Yosys didn't know the input/output pins. One way that worked was providing the black box model for lut3, but using hierarchy -generate also worked to specify the in/out ports.