faq 107754740

Advice on parsing/analyzing large events files

by Larry Walker on 2017-02-08 16:40:29

---

Hello everyone,

The situation – I've been learning MATSim for the past few weeks and am getting to the point where I'm ready to start building a simulation approaching production-level. For me, that would mean, at first anyway, simulating vehicle traffic traveling over a large urban area with probably something like 5 million agents. I realized after playing with a simulation involving 100,000 agents that the unzipped events file was already 4GB, and so I'm probably looking at having unzipped events files in the vicinity of 100GB+ (right?).

The question – Does anyone have any tips on how to do custom analysis on data generated by MATSim for a large simulation? I'm thinking relatively simple things – like, for example, a graph by time of day for average speed on a specific link. I figured I could extract this info from the unzipped events file, but I have no idea what a reasonable approach is for parsing a file that large. (For the record, I have experience with VBA, R, and pgSQL, but would be happy with ideas in any language.) Given that some of the examples in the book (i.e. Berlin) use a similar geographic area and millions of agents, I figured this is a solved problem. I'd appreciate some pointers.

Thanks in advance. 

---

Comments: 1

---

Re: Advice on parsing/analyzing large events files

by Joschka Bischoff on 2017-02-08 16:46:11

Hi Larry,

I would recommend you to use the post processing event handling tools MATSim provides out of the box.

For a tutorial on how to use them, see the MATSim tutorial:

https://isis.tu-berlin.de/course/view.php?id=8037

section Analysing MATSim output data (II)

The direct link to the code is here.

Good luck

Joschka