you know, you could also either include a lossy copy next to the lossless ones, then rsync only lossy extensions, or, if that pollutes your collection, have a separate but identically-structured directory tree, where all your lossless files have lossy copies. Then, you can rsync both folders (send-only) to your single remote (lossy extensions only).
but, yeah, Git really isn't the tool for this, agreed.
good looking out.