I don’t like how they are all using him

But at the same time they aren't..
Like... the castle wall... he did his work but after being fired he thought he did a bad job...
With mining, she never forced him to do any of that... she just wanted help mining..

The porridge was just that... porridge, he didn't mean to do anything funny to it

Ya. He just does things excessively and doesn’t know where to stop. Like with the magic crystals, she gave him all the money except one coin. He wasn’t forced to do anything or persuaded. Of course they are thing to use him in their plan, but they likely won’t force him so I’m fine with it. He seems content with any work too (⌒▽⌒)