Oh yes, you are completely right, and I loved it for its difficulty. The game actually has a set path it wants to make you go through from a leveling and difficulty perspective, even though you don't have to go though it that way. To do otherwise though you would have to be amazing at battle system, and be prepared for long drawn out fights.
Sadly the game is not that good at directing you towards the appropriate difficulty areas, so it takes a bit of saving, then dipping your toe in the water with a area to find out, but to players who don't have the time for that trial and error approach, I wouldn't hesitate to recommend just looking at the index for a walkthrough for the game just to see the order it is recommend to tackle the areas, quests, and instances of map.
Even with all of this, the other thing to remember is to heavily specialize and balance your characters, a Jack of all trades, or double up of profession in the characters will spell doom for the party. If one guy is a pyromancer? pump all ability points into Pyro and Int first.
It does get better.... (for you I hope!!)