Input file size

Algorithm

Records are extracted from text files with delimiter set to 2 newlines. Every record is processed with a few regexp (Java regexp). Records are grouped by key. Grouped records are sorted in order to properly calculate time between them. The output is stored in HDFS as text file.

Environment

Spark job was executed with Job Server, non-blocking Ratpack Framework application.

PROCESSING TIME