Japanese and r/l mixup

Was thinking about Lost in Translation again and realized that while in the movies such as this Japanese often say [r] instead of [l] and [l] instead of [r], in real life I have only heard them say [r] instead of [l]. Some books say that in Japanese [r] and [l] are allophones of one phoneme, but I have an impression that I've never heard them pronounce it as [l], only either as a "normal" trilled alveolar [r] or as a more [d]-like alveolar flap. What's the deal with this? Do the Japanese really say [l] while speaking Japanese?