custom dataset error when second epoch ?

[iamuspace]

I am trying to make my custom dataset as AND gate.

Dataset is below :
0,0 = 0
0,1 = 0
1,0 = 0
1,1 = 1

My code runs OK for a first epoch, but error at second epoch.

What is the solution for this error ?

--- std out ---
new AndSampler()
new AndGateDataset()
pre fit
sample() call
rad size=4
sampler list=[[0, 1, 2, 3]]

Training: 100% |========================================|
sample() call
rad size=4
sampler list=[[0, 1, 2, 3]]
Exception in thread "main" java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:100)
at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:106)
at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:302)
at java.base/java.util.Objects.checkIndex(Objects.java:385)
at java.base/java.util.ArrayList.get(ArrayList.java:427)
at ai.djl.ndarray.NDList.head(NDList.java:232)
at ai.djl.training.dataset.Batch.split(Batch.java:208)
at ai.djl.training.EasyTrain.trainBatch(EasyTrain.java:91)
at ai.djl.training.EasyTrain.fit(EasyTrain.java:58)
at test.djl.dataset.AndTest.main(AndTest.java:72)

--- main() ---

	public static void main(String[] args) throws IOException, TranslateException {		
		long inputSize = 2;
		long outputSize = 1;
		
		SequentialBlock block = new SequentialBlock();		
		block.add(Blocks.batchFlattenBlock(inputSize));
		block.add(Linear.builder().setUnits(outputSize).build());
		
		Model model = Model.newInstance("and");
		model.setBlock(block); 		
		
		DefaultTrainingConfig config = new DefaultTrainingConfig(Loss.sigmoidBinaryCrossEntropyLoss())
			.addEvaluator(new Accuracy()) 
			.addTrainingListeners(TrainingListener.Defaults.logging());

		int batchSize = 4; 
		Trainer trainer = model.newTrainer(config);
		trainer.initialize(new Shape(batchSize,2));
		
		Dataset dataset = new AndGateDataset.Builder().build(); 
		
		System.out.println("pre fit");
		int epoch = 2;
		EasyTrain.fit(trainer, epoch, dataset, null);
	}

--- AndGateDataset.java ---

public class AndGateDataset extends RandomAccessDataset {
	private final List<Record> records; 

    public AndGateDataset(Builder builder) {
		super(builder); 
		System.out.println("new AndGateDataset()");
		this.records = builder.records; 
    }
	
	@Override
	public Record get(NDManager manager, long index) throws IOException {
		Record record = this.records.get(Math.toIntExact(index)); 
		return record; 
	}

	@Override
	protected long availableSize() {
		return this.records.size(); 
	}
	
	public static final class Builder extends BaseBuilder<Builder> {
		public Builder() { }
		List<Record> records = new ArrayList<>(); 
		
		@Override
		protected Builder self() {
			return this; 
		}
		
		AndGateDataset build() throws IOException {
			NDManager manager = NDManager.newBaseManager();
			
			NDArray data = manager.create(new float[]{0,0});
			NDArray label = manager.create(new float[]{0});
			Record andRecord = new Record(new NDList(data), new NDList(label)); 
			this.records.add(andRecord);
			
			data = manager.create(new float[]{1,0});
			label = manager.create(new float[]{0});
			andRecord = new Record(new NDList(data), new NDList(label)); 
			this.records.add(andRecord);
			
			data = manager.create(new float[]{0,1});
			label = manager.create(new float[]{0});
			andRecord = new Record(new NDList(data), new NDList(label)); 
			this.records.add(andRecord);
			
			data = manager.create(new float[]{1,1});
			label = manager.create(new float[]{1});
			andRecord = new Record(new NDList(data), new NDList(label)); 
			this.records.add(andRecord);
			
			Sampler sampler = new AndSampler(this.records); 
			this.setSampling(sampler);
			return new AndGateDataset(this); 
		}
	}
	
	@Override
	public void prepare(Progress prgrs) {		
	}
}

--- AndSampler.java --- 

public class AndSampler implements Sampler {
	List<Record> records; 
	public AndSampler(List<Record> records)
	{
		System.out.println("new AndSampler()");
		this.records = records; 
	}
	
	@Override
	public Iterator<List<Long>> sample(RandomAccessDataset rad) {
		System.out.println("sample() call ");
		System.out.println("rad size="+rad.size());
		
		List<List<Long>> list = new ArrayList(); 
		ArrayList<Long> batch1 = new ArrayList();
		batch1.add(0L);
		batch1.add(1L);
		batch1.add(2L); 
		batch1.add(3L); 
		list.add(batch1); 

		System.out.println("sampler list="+list);
		return list.iterator(); 
	}

	@Override
	public int getBatchSize() {
		return this.records.size(); 
	}	
}

```

[frankfliu]

Based on your stacktrace, it seems your dataset returned an empty Batch, I think you have to debug your dataset code.

[iamuspace]

I haven't been able to solve the problem yet, and the problem is very, very difficult. It would be nice to have a tutorial or sample code to create a custom dataset for a very simple dataset.

[frankfliu]

Can you create check in your code in a repo that I can test your code?

[iamuspace]

Thank you so much frank.
I made new my repository as fast DJL tutorial.

link is
https://github.com/iamuspace/djl-tutorial.git

[frankfliu]

@iamuspace
In your [AndGetDataset.get(NDManager manager, long index)](public Record get(NDManager manager, long index)) function, you returned the NDArray of the dataset directly.

The NDArray returned from this function will be closed automatically after being used. That's why you see the exception in 2nd epoch.

There are two ways to resolve this issue:

1. duplicate the NDArray before add to Record
2. Create a wrapper class, override close() method to prevent DataIterator to close it.

[iamuspace]

WOW ! bug fixed !
Thank you so much frank !

By the way,
Even in repeated epochs,
the NDArray of Record is newly recreated each time.
Is this intended to conserve memory so that it can be processed even when the dataset is very large?

[adinath1233]

@frankfliu I have question can you tell me is there any specific reason you selected batch size as 4 ?