@oliveryuan,
No, it's not difficult at all.
Looking at the specs for your motherboard, it supports up to 64GB of RAM. From your OP, you have two free slots to install another two sticks of RAM. So yes, you could install more RAM.
As said above, while each pair in a set should match, the two sets can be different as long as it's the same type of RAM (in this case DDR4 2666/2400/2133 MHz memory modules).
While I don't have the same motherboard as you in my current desktop machine; I do have two different speeds installed being two DDR3 1333MHz and two DDR3 1666MHz RAM modules. The board simply just defaults to the lowest speed (in this case 1333MHz) for both sets - as it says in the motherboard manual if you have RAM modules of different speeds installed. Which in my case since the board only supports up 1333Mhz RAM speeds, then it's still operating at maximum speed the board can support anyway.
https://www.gigabyte.com/au/Motherboard/B365-HD3-rev-10/sp#sp