Trouble getting quicksync to work with jellyfin

Help
1

I set up a server with an intel 14500 running nixos 23.11. Jellyfin works fine, and vaapi works, but quicksync keeps giving the same error:

[AVHWDeviceContext @ 0x1f86fc0] Error initializing an MFX session: -3.
Device creation failed: -1313558101.
Failed to set value 'qsv=qs@va' for option 'init_hw_device': Unknown error occurred
Error parsing global options: Unknown error occurred

In my config I set the following options from the wiki to enable intel hardware acceleration.

  hardware.opengl = {
    enable = true;
    extraPackages = with pkgs; [
      intel-media-driver
      intel-compute-runtime # OpenCL filter support (hardware tonemapping and subtitle burn-in)
      vaapiVdpau
      libvdpau-va-gl
    ];
  };
  environment.sessionVariables = { LIBVA_DRIVER_NAME = "iHD"; };

I tried enabling quicksync both with and without low-power encoding and decoding and jellyfin but neither works, I am using linux kernel 6.7(Latest version compatible with zfs).

Edit:
I seem to have gotten it working with onevpl, at the time of writing this isn’t available in any stable or unstable branch of nixos, having just been merged into staging-next pr.

I added it to my opengl config as such:

  hardware.opengl = {
    enable = true;
    extraPackages = with pkgs; [
      intel-media-driver
      intel-compute-runtime # OpenCL filter support (hardware tonemapping and subtitle burn-in)
      (pkgs.callPackage ../../pkgs/onevpl-intel-gpu.nix { })
    ];
  };

And recompiled jellyfin-ffmpeg to use libvpl

    config.packageOverrides = prev: {
      jellyfin-ffmpeg = prev.jellyfin-ffmpeg.overrideAttrs (old: rec {
        configureFlags =
          # Remove deprecated Intel Media SDK support
          (builtins.filter (e: e != "--enable-libmfx") old.configureFlags)
          # Add Intel VPL support
          ++ [ "--enable-libvpl" ];
        buildInputs = old.buildInputs ++ [
          # VPL dispatcher
          pkgs.unstable.libvpl
        ];
      });
    };

Note that libvpl isn’t available in 23.11 so I used the unstable branch for this package.

2

Here’s a slightly more compact configurations to get jellyfin with QSV on Arc-series GPUs:

{ config, pkgs, ... }:
let
  jellyfin-ffmpeg-overlay = (final: prev: {
    jellyfin-ffmpeg = prev.jellyfin-ffmpeg.override {
      ffmpeg_6-full = prev.ffmpeg_6-full.override {
        withMfx = false;
        withVpl = true;
      };
    };
  });
  unstable = (import <unstable> {
    config.allowUnfree = true;
    overlays = [ jellyfin-ffmpeg-overlay ];
  });
in
{
  services.jellyfin = {
    enable = true;
    package = unstable.jellyfin;
  };

  boot.kernelPackages = pkgs.linuxPackages_latest;

  environment.variables = {
    NEOReadDebugKeys = "1";
    OverrideGpuAddressSpace = "48";
  };

  systemd.services."jellyfin".environment = {
    NEOReadDebugKeys = "1";
    OverrideGpuAddressSpace = "48";
  };

  hardware.opengl = {
    enable = true;
    extraPackages = with pkgs; [
      intel-media-driver
      intel-compute-runtime
      unstable.onevpl-intel-gpu
    ];
  };
}

The environment variables are from https://github.com/intel/compute-runtime/issues/710#issuecomment-1972684586
Also, jellyfin docs claim that their ffmpeg versions 6+ have OneVPL enabled, which NixOS is not respecting. Maybe the jellyfin-ffmpeg package needs to be updated to match (or change the base ffmpeg-full package to enable VPL instead of MFX).

1 Like