“Waste” is probably the wrong word. Even moments that would not be considered wasted by most people, he saved for Torah study.
For example, he did not visit the cemetery on the yahrzeit, the anniversary of the death, of his father. (I was told this when his mother was still alive; presumably, he did not visit her grave, either.)
The person who told me this was Rabbi Noah Heisler, a rabbinic judge in Jerusalem.
During the 1970s, I drove Rabbi Heisler and his father to visit three rabbinic eminences during the intermediate days of Passover and of Sukkot: Rabbi Yosef; Rabbi Yosef Shalom Eliashiv, who died a year ago at 102; and Rabbi Bezalel Zolti, the chief rabbi of Jerusalem, who died at 61 in 1982.