That can be an effective tool, if done correctly. It sounds like you're on the right path. Use the LLM as a mirror to your own speculations and guide the discussion to the finer details. You are in a way talking to yourself much like someone would be writing in a journal, however it's more organized and faster. Just be wary of the hallucinations. I've found modifying the system prompt for replies to be short and direct unless asked to go into detail along with a short summary of your overall goals to be helpful in keeping it on track. Also regularly summarize the current session once it gets long and start a new one with the summary to avoid context wandering.
I also found Claude was better for me than Gemini and ChatGPT, but it could depend on the subject (I'm doing fiction).
One example for the detail work is something I did recently. Hammering out the technology and physics aspects of the novel's universe. We went back and forth, played with ideas, until I had a much better picture of what I want it to be like. And I had to steer Claude away a few times from diverting to other things, even with my system prompt to stay on topic. It's just what LLMs do.
At some point step away from the LLM and finish the writing yourself. It's okay to use it even for revision work, but at that point have it develop lists and suggestions for you to work with, not write the text. You'll avoid a lot of headaches later trying to "humanize" it.