Tools and other software Media-Buddy - the MKV Buddy successor. (7 Viewers)

Lehmden

Retired Team Member
  • Premium Supporter
  • December 17, 2010
    12,626
    4,028
    Lehmden
    Home Country
    Germany Germany
    Hi.

    The music scraper is coming along nicely. It’s looking like it will turn out to be many times better than I ever expected.

    V2_327.jpg


    In principle, you can feed the scraper a collection of unsorted .mp3, .flac, .ogg, or similar files. As long as there is some indication of which track or album a file belongs to—such as its filename or ID3 tags—the scraper will (almost always) correctly identify it, apply the appropriate tags, rename the file according to a configurable scheme, and download the corresponding lyrics. Much like with TV shows or movies, you can also provide a MusicBrainz ID to assist with identification. The scraper can also retrieve fan art and metadata for both albums and artists—all of which is fully configurable. Optionally, song lyrics, album information, and artwork can be embedded directly into the music files, similar to how most dedicated external tagging programs operate. Furthermore, if configured to do so, the scraper can create folders for albums and/or artists, and store artist-related information in Kodi’s central artist artwork directory (either in addition to or instead of embedding it in the files).

    V2_328.jpg


    And all of this happens fully automatically, without requiring any user intervention. Naturally, there is also an option for manual adjustments, allowing you to correct any errors made by the automated process. That said, in recent tests involving over 100 different audio files, the scraper achieved a 100% success rate—a significant improvement from the initial 60–70% range. The extra effort has truly paid off, though, of course, no absolute guarantee can be given.

    While working on the music scraper (a project still in progress), the idea emerged to potentially include song lyrics as subtitles in music videos—essentially converting .lrc files into .srt files. This strikes me as a great idea; it would allow users to choose whether or not to display the lyrics as subtitles when playing music videos. This feature is not yet complete, but it is being implemented.
     

    Attachments

    • de6aa864-7274-4a13-9910-5f149d558a4b.zip
      214 bytes

    Lehmden

    Retired Team Member
  • Premium Supporter
  • December 17, 2010
    12,626
    4,028
    Lehmden
    Home Country
    Germany Germany
    Hi.

    The scraper is completely finished and has turned out really well. I never expected it to be able to generate such high-quality metadata tags in a fully automated way. It handles unusual cases—like standalone tracks or duets—with ease, and it’s also unfazed by "dangerous" files; for instance, I discovered some files that contained JavaScript embedded within the lyrics. While JavaScript cannot be executed directly from a music file, it certainly has no business being in a set of song lyrics, in my opinion. In any case, such content—as well as remnants of HTML—is now cleaned out, and the "correct" lyrics are inserted as metadata tags (provided the software is configured to do so). I’m not sure to what extent tools like CULyrics can make use of embedded lyrics, but if the external lyric files were ever to go missing, you could simply extract them again from the metadata.

    V2_341.jpg


    Artist and album fan art, along with metadata, are now stored—optionally—both directly alongside the music files and in a central "artist artwork" folder. This entire process happens automatically, without any manual intervention, and achieves an unexpectedly high success rate.

    With that, the "workhorses" of the music-processing system are complete. Next up is the "ringmaster"—the Music Assistant. It will autonomously decide whether a music file should be copied as-is or re-encoded, and will then trigger the scraper to transform a chaotic jumble of files into organized, neatly tagged albums and individual tracks. Once the final manual refinement and quality control are finished, the music component will be complete. Slowly but surely, I can finally see the light at the end of the tunnel...
     

    Lehmden

    Retired Team Member
  • Premium Supporter
  • December 17, 2010
    12,626
    4,028
    Lehmden
    Home Country
    Germany Germany
    Hi.

    I spent some more time refining the scraper yesterday afternoon and this morning. Various edge cases kept popping up that weren’t yet handled automatically—especially when processing unsorted, untagged individual files in a batch. But by around midday, I couldn’t find a single music file that hadn’t been processed correctly. It works just as well for private, never-released CDs recorded 30 years ago as it does for audio tracks from YouTube videos by unknown artists... The assistant is also working perfectly.

    V2_369.jpg


    There are separate settings for re-encoding lossy formats (such as .mp3, .ogg, etc.) and lossless formats (such as .flac, .ape, etc.). Additionally, there is a whitelist containing codecs that should not be re-encoded at all. The assistant uses this whitelist to determine whether a track requires re-encoding, while also taking into account the settings you have configured. For instance, an .ogg file might be converted to .mp3, while an .ape file could be converted to .flac. The quality settings are intentionally kept simple—much like those for video. Note that for lossless formats, you cannot actually adjust the "quality" (since it is, by definition, lossless); instead, these settings control the compression rate. A higher compression rate results in a smaller file size—without altering the actual audio track—though it also requires more time and CPU power for encoding and decoding. Furthermore, you can increase the volume of tracks (only when re-encoding), which is useful if an album sounds too quiet compared to others. In such cases, it is best to process the albums individually; otherwise, every track will be boosted by the same amount. Once all music files have been processed, the assistant automatically runs a scraper on the output folder. In most cases, the files are instantly renamed, tagged, and populated with fan art and lyrics—ready to be added to your personal collection.

    With that, I was finally able to begin the post-processing of the music.

    Take, for example, an audio track from a YouTube music video by a former "The Voice Kids" contestant: "I Follow" by "Loi." Since this track cannot be assigned to any specific album, the scraper is unable to retrieve an album cover. However, the artist.nfo file and the fan art were generated without any issues. Furthermore, the file was converted from .ogg to .mp3 according to my settings, renamed appropriately, and both the local and central artist folders were created and populated with content. In short, everything that could be handled automatically was taken care of (correctly).

    V2_361 - Kopie.jpg


    There isn't much data available, but that’s simply the nature of the situation—artist information isn't displayed for these tracks. For a "standard" track from a "standard" album, the album cover is typically embedded into each track, just as it would be with any other tagging tool. However, since there is no album in this case, there is no artwork. I can manually add details like the year or genre, as well as the cover art. To make the search easier, there’s an "Image Search" button; it opens Google Images using the artist's name as the search term. Within seconds, I was able to find and load this image:

    images (1).jpg


    This is what it then looks like in the GUI:

    V2_367.jpg


    So, this track now has its cover art, too. You can see the benefit of this immediately when playing it back as a test from within Media-Buddy.

    V2_363.jpg


    The lyrics are displayed in sync. Without the track cover, there would just be a black window. Speaking of lyrics: they don’t always match up, simply because there can be so many different versions of a single song. In the case of "I Follow" (above), everything lined up perfectly—no adjustments were needed. But in the next example—which uses the audio track from the official music video—the lyrics appear far too early, as the video includes a long intro.

    V2_368.jpg


    This allows you to shift the display of the song lyrics forward or backward. In most cases, it takes no more than two attempts to get the lyrics perfectly synchronized with the song.

    V2_362.jpg


    This gets you quite a long way without having to edit the text files themselves. Furthermore, the modified lyrics—or, more precisely, the lyrics with updated timestamps—are written back into the metadata tags. A similar feature is also being integrated for music videos, allowing the lyrics to be saved as subtitles.

    Next comes post-processing at both the album and artist levels; the artist-level post-processing also applies to music videos. Ultimately, both music videos and music tracks share the same central artist artwork folder and the same .nfo structure. With a bit of luck, the music component will be finished this week, and the updated source code will be uploaded to GitHub.
     

    Users who are viewing this thread

    Top Bottom