Xtykutl

I am processing a large number of records (CDRS) that are essentially (who, where, how much), to save space I use a lookup to map the strings into integer and aggregate the traffic on a map of maps (who maps to a map (where maps how much)

type CDR      = (String, String, Int)

type Lookup   = scala.collection.mutable.HashMap[String, (Int, Float)]

type Traffic  = scala.collection.mutable.HashMap[Int,scala.collection.mutable.HashMap[Int,Int]]enter code here

I have found a strange behavior, when I build the lookup tables in advance the code runs as expected, however when I start processing and build the maps on the fly it slows down as it processes the records.

I use the same function to build the lookup tables for this comparison. I essentially check if the code for the lookup is there, if not i create a new entry (it is a mutable map), like this:

def index(id: String, map: Lookup, reverse: Reverse): Int = {

    if (map.contains(id)) {

        map(id)._1

    } else {

        val number = if (map.keys.size == 0) 0 else reverse.keys.max + 1

        reverse += ( number -> id)

        map += (id -> (number, 0.toFloat))

        number   

    }

}

Am I missing something here?
EDIT----> I can no longer reproduce the slowdown. I will assume I was either too tired or dumber than usual. Running time now seems to be same as I expected to be.

What is mapCellRvs? Default scala Map's .size (and .keys.size, which is the same thing) simply counts all elements by scanning them linearly.

Try replacing mapCellRvs.keys.size == 0 with mapCellRvs.isEmpty ...

Also, reverse.keys.max is linear as well. You may want to just remember the max somewhere separately, rather than compute it every time.