I see two different possible arguments from your position:

1. The file was provided to you for free. Since there's no profit involved in that transaction, it shouldn't be illegal to make a DRM-free copy of the file.

2. Shouldn't you be able to listen to the file any time you want, without having to keep to the library's timetable?

I think both arguments can be countered with the printed book metaphor:

What if you checked a printed book out of the library, then photocopied every page of that book before returning it to the library on its due date? This doesn't take any profit away from the library or publisher, and (like the DRM removal) requires that you go to some trouble on your part. It allows you to consume the book on your own time.

I think, whether you're talking about a book or an audio file, it's still considered breaking copyright law. And by the way, copyright law doesn't care whether "you would have bought it" or not.