Apache Spark Code Review - PySpark Memory Limits (less OOMS) & Py3 cleanup.
Continuing the series I'll do a live code review of some of the changes going into Spark. This week we looked at a PR proposing to use rlimits to help fix out of memory issues with PySpark (super exciting) - https://github.com/apache/spark/pull/21977#pullrequestreview-143243633 & important Py3 cleanup work (PySpark work in Py3 but there's a few corner cases where we were doing odd things) - https://github.com/apache/spark/pull/20838 :)
Continuing the series I'll do a live code review of some of the changes going into Spark. This week we looked at a PR proposing to use rlimits to help fix out of memory issues with PySpark (super exciting) - https://github.com/apache/spark/pull/21977#pullrequestreview-143243633 & important Py3 cleanup work (PySpark work in Py3 but there's a few corner cases where we were doing odd things) - https://github.com/apache/spark/pull/20838 :)