The confusion about memory occupation

[SRQMAX]

I use Linux system to mount the SD card of exFAT file system. I found that when I was browsing through the files in the SD card The memory consumption of the mount.exfat process is increasing. The most important thing is that this memory cannot be released unless the SD card is umounted.
Prepare 10000 photos for each folder, execute ls command to browse them.About 7 MB of memory is used for every 10000 files.The memory occupied by these browsing files cannot be released until SD is unloaded. Is this normal?
I use the echo 3 > /proc/sys/vm/drop_caches command to release the cache.